[2025-05-15 09:40:26.584260] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9062-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.584273] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9062-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552599, proxy_sessid=0) [2025-05-15 09:40:26.584295] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9062-0-0] [lt=12][errcode=0] server close connection(sessid=3221552599, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.584307] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9062-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552599, proxy_sessid=0) [2025-05-15 09:40:26.584315] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9062-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:26.584325] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9062-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552599, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:26.584371] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221552599, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302026583983, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.584405] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] kill and revert session(conn.sessid_=3221552599, proxy_sessid=0, ret=0) [2025-05-15 09:40:26.584417] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221552599, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302026583983, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:26.584432] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552599) [2025-05-15 09:40:26.584449] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552599, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:26.585748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.586898] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552600, support_ssl=false) [2025-05-15 09:40:26.586946] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552600, sess.client_addr_="172.22.0.1:42116") [2025-05-15 09:40:26.586964] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552600, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.587117] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:26.587148] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:26.587162] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:26.587183] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:26.587191] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:26.587204] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:26.587218] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587235] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552600, proxy_sessid=0) [2025-05-15 09:40:26.587239] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552601, support_ssl=false) [2025-05-15 09:40:26.587252] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=7][errcode=0] server close connection(sessid=3221552600, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587267] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552600, proxy_sessid=0) [2025-05-15 09:40:26.587274] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:26.587266] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=33] new mysql sessid created(conn.sessid_=3221552602, support_ssl=false) [2025-05-15 09:40:26.587282] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9023-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552600, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:26.587298] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] sm conn init succ(conn.sessid_=3221552601, sess.client_addr_="172.22.0.1:42130") [2025-05-15 09:40:26.587316] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552602, sess.client_addr_="172.22.0.1:42140") [2025-05-15 09:40:26.587323] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552600, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302026586104, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.587345] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552600, proxy_sessid=0, ret=0) [2025-05-15 09:40:26.587342] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552602, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.587333] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552601, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.587354] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552600, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302026586104, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:26.587373] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552600) [2025-05-15 09:40:26.587387] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552600, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:26.587503] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:26.587500] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552603, support_ssl=false) [2025-05-15 09:40:26.587544] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552603, sess.client_addr_="172.22.0.1:42152") [2025-05-15 09:40:26.587547] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:26.587557] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552603, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.587579] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=32][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:26.587600] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:26.587603] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:26.587622] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:26.587624] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:26.587630] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:26.587642] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:26.587655] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:26.587654] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587669] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:26.587676] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:26.587673] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552601, proxy_sessid=0) [2025-05-15 09:40:26.587689] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587703] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552602, proxy_sessid=0) [2025-05-15 09:40:26.587705] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=14][errcode=0] server close connection(sessid=3221552601, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587719] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552601, proxy_sessid=0) [2025-05-15 09:40:26.587726] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=12][errcode=0] server close connection(sessid=3221552602, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587733] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:26.587737] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552602, proxy_sessid=0) [2025-05-15 09:40:26.587748] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:26.587742] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9032-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552601, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:26.587759] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9063-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552602, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:26.587792] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552601, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302026587162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.587809] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:26.587819] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552601, proxy_sessid=0, ret=0) [2025-05-15 09:40:26.587832] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:26.587828] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552601, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302026587162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:26.587845] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552601) [2025-05-15 09:40:26.587848] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:26.587866] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:26.587858] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552602, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302026587162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.587861] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552601, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:26.587875] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:26.587885] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:26.587880] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552602, proxy_sessid=0, ret=0) [2025-05-15 09:40:26.587894] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552602, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302026587162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:26.587900] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587914] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552602) [2025-05-15 09:40:26.587914] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552603, proxy_sessid=0) [2025-05-15 09:40:26.587932] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=7][errcode=0] server close connection(sessid=3221552603, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:26.587928] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552602, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:26.587945] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552603, proxy_sessid=0) [2025-05-15 09:40:26.587954] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:26.587961] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9024-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552603, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:26.588006] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552603, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302026587162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:26.588029] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552603, proxy_sessid=0, ret=0) [2025-05-15 09:40:26.588070] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552603, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302026587162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:26.588090] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552603) [2025-05-15 09:40:26.588101] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552603, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:26.590714] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:113) [4578][T1002_TimerWK1_SSTableGC][T1002][YB427F000001-000635291F6C666F-0-0] [lt=27] SSTableGCTask(cost_ts=13735) [2025-05-15 09:40:26.595858] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.598128] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:26.598848] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:26.598879] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302026598841) [2025-05-15 09:40:26.598897] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302026398781, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:26.598929] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12328847610, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.598949] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.598974] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.598990] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.599001] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.599016] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12328847610, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.599029] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.599043] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.599054] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:26.599063] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:26.599075] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.599085] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.599094] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026598910) [2025-05-15 09:40:26.605971] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.611007] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.611028] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:26.611049] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:26.611069] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:26.611082] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:26.611096] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.611105] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.611117] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:26.611130] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302026610973, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.611145] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:26.611154] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.611231] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:26.611248] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:26.611263] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:26.611275] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:26.616084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.619190] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [4090][Blacklist][T0][Y0-0000000000000000-0-0] [lt=58] blacklist_loop exec finished(cost_time=19, is_enabled=true, send_cnt=0) [2025-05-15 09:40:26.620650] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:26.620695] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302026620642) [2025-05-15 09:40:26.620712] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302026420595, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:26.620749] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12328892084, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:26.620771] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.620797] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:26.620817] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.620833] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.620850] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.620861] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.620872] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026620728) [2025-05-15 09:40:26.620947] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12328719591, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:26.620969] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.620989] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:26.621005] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.621016] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.621028] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.621038] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.621050] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026620932) [2025-05-15 09:40:26.621966] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EAAB0-0-0] [lt=29][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302026193170}) [2025-05-15 09:40:26.622854] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=6] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:26.623346] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.623367] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17648352, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.623383] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17648352, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.623396] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:26.623407] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=17648352, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:26.623421] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:26.623436] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:26.623450] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.623463] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.623472] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.623486] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.623514] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026623512, replica_locations:[]}) [2025-05-15 09:40:26.623531] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:26.623554] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:26.624750] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.624771] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.624786] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.624807] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.624820] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.624831] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.624845] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.624858] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026624857, replica_locations:[]}) [2025-05-15 09:40:26.624919] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=17646763, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=171, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:26.626196] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.637769] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.647883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.657995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.668110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.678223] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.688337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.693295] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:26.698183] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:26.698443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.698923] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12328946783, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.698949] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.698972] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.698989] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.698997] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.699012] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12328947798, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.699025] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.699039] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.699050] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:26.699059] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:26.699071] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.699080] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.699092] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026698902) [2025-05-15 09:40:26.708572] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.711002] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.711024] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:26.711043] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:26.711062] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:26.711076] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.711086] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.711097] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:26.711108] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302026710978, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.711122] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:26.711134] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.711202] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:26.711215] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:26.711223] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:26.711234] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:26.713375] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=58] table not exist(tenant_id=1, database_id=201001, table_name=__all_mview, table_name.ptr()="data_size:11, data:5F5F616C6C5F6D76696577", ret=-5019) [2025-05-15 09:40:26.713406] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_mview, ret=-5019) [2025-05-15 09:40:26.713417] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_mview, db_name=oceanbase) [2025-05-15 09:40:26.713430] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_mview) [2025-05-15 09:40:26.713442] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:26.713451] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:26.713466] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_mview' doesn't exist [2025-05-15 09:40:26.713477] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:26.713495] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:26.713505] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=10][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:26.713515] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:26.713525] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:26.713534] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:26.713543] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:26.713562] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:26.713572] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:26.713583] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:26.713593] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:26.713602] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4, ret=-5019) [2025-05-15 09:40:26.713613] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:26.713620] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:26.713637] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:26.713656] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:26.713665] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:26.713673] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:26.713689] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:26.713703] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_TntSharedTimer][T1][YB427F000001-000635291C5C638B-0-0] [lt=0][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5019, ret=-5019) [2025-05-15 09:40:26.713714] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:26.713725] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:26.713736] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:26.713745] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302026713109, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:26.713755] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:26.718694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.720721] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12328992145, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:26.720747] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.720771] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:26.720789] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.720800] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.720815] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.720826] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.720837] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026720702) [2025-05-15 09:40:26.720994] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:26.721019] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302026720987) [2025-05-15 09:40:26.721035] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302026520943, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:26.721058] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12328820707, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:26.721073] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.721092] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:26.721107] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.721119] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.721133] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.721143] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.721150] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026721047) [2025-05-15 09:40:26.725052] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.725076] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17546649, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.725093] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17546649, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.725120] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.725135] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.725147] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.725162] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.725179] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026725178, replica_locations:[]}) [2025-05-15 09:40:26.725197] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:26.725219] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:26.726385] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.726407] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.726422] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.726442] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.726455] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.726465] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.726479] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.726497] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026726496, replica_locations:[]}) [2025-05-15 09:40:26.726560] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=17545122, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=172, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:26.728805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.738917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.740776] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3955][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:26.749028] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.751696] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.751713] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:26.751723] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:26.751739] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:26.751753] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:26.751788] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:26.759138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.769252] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.779361] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.789472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.794512] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.798264] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:26.798289] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:26.798302] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:26.798963] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:26.798982] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302026798960, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302026798960, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:26.799033] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:26.799057] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302026799027) [2025-05-15 09:40:26.799070] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302026598909, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:26.799100] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12329047744, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.799117] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.799137] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.799152] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.799162] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.799177] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12329047744, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.799189] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.799203] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.799213] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:26.799222] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:26.799234] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.799243] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.799253] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026799083) [2025-05-15 09:40:26.799591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.809699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.809717] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-15 09:40:26.811028] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.811049] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:26.811063] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:26.811077] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.811092] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:26.811105] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.811116] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.811127] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:26.811138] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302026811010, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.811152] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:26.811160] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.811212] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:26.811224] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:26.811231] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:26.811239] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:26.819419] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:26.819820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.820766] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:26.820786] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302026820761) [2025-05-15 09:40:26.820911] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=126][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302026620727, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:26.820934] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12329092087, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:26.820950] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.820970] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:26.820985] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.820998] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.821014] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.821024] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.821030] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026820921) [2025-05-15 09:40:26.821054] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1001, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:26.821087] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=12328920651, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:26.821104] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.821121] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:26.821136] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.821147] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.821162] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.821174] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.821180] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026821075) [2025-05-15 09:40:26.821706] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302026821703, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007611, add_bytes: 0 [2025-05-15 09:40:26.822948] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:26.826630] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4462][T1001_TimerWK2_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-05-15 09:40:26.826649] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4462][T1001_TimerWK2_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=16] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:26.826682] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.826700] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17445014, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.826714] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17445014, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.826727] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.826746] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.826759] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.826770] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.826785] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.826801] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026826799, replica_locations:[]}) [2025-05-15 09:40:26.826817] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:26.826840] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:26.827591] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=36] PNIO [ratelimit] time: 1747302026827588, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-05-15 09:40:26.828005] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.828026] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.828041] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.828061] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.828075] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.828085] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.828099] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.828112] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026828112, replica_locations:[]}) [2025-05-15 09:40:26.828170] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=17443511, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=173, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:26.829838] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, table_name.ptr()="data_size:25, data:5F5F616C6C5F6469736B5F696F5F63616C6962726174696F6E", ret=-5019) [2025-05-15 09:40:26.829870] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, ret=-5019) [2025-05-15 09:40:26.829885] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_disk_io_calibration, db_name=oceanbase) [2025-05-15 09:40:26.829900] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_disk_io_calibration) [2025-05-15 09:40:26.829916] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:26.829925] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=8][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:26.829932] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.829942] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_disk_io_calibration' doesn't exist [2025-05-15 09:40:26.829952] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:26.829959] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:26.829970] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=10][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:26.829983] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:26.829995] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:26.830007] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:26.830016] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:26.830039] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:26.830052] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:26.830064] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:26.830077] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:26.830089] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA", ret=-5019) [2025-05-15 09:40:26.830103] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:26.830112] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:26.830131] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:26.830152] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:26.830164] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=12][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:26.830172] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:26.830205] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA""}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:26.830218] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6182-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.830230] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [3955][TimerWK2_ServerGTimer][T0][YB427F000001-000635291F3C6182-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA""}, aret=-5019, ret=-5019) [2025-05-15 09:40:26.830244] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA") [2025-05-15 09:40:26.830254] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:26.830270] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:26.830280] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302026829572, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA") [2025-05-15 09:40:26.830292] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:26.830303] WDIAG [COMMON] parse_calibration_table (ob_io_calibration.cpp:908) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, sql_string=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA") [2025-05-15 09:40:26.830365] WDIAG [COMMON] read_from_table (ob_io_calibration.cpp:773) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] parse calibration data failed(ret=-5019) [2025-05-15 09:40:26.830380] WDIAG [SERVER] refresh_io_calibration (ob_server.cpp:3859) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] fail to refresh io calibration from table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:26.830392] WDIAG [SERVER] runTimerTask (ob_server.cpp:3849) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] ObRefreshIOCalibrationTimeTask task failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:26.833696] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4624][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=26] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1a903de2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:26.840045] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.850162] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.853985] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=29] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:26.854015] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:26.854029] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:26.854042] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:26.854056] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:26.854065] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:26.854079] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:26.854089] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:26.854098] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:26.854105] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=6][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:26.854114] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:26.854124] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:26.854133] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:26.854142] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:26.854158] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:26.854168] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:26.854178] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:26.854189] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:26.854197] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:26.854208] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=9][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:26.854216] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:26.854234] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:26.854252] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:26.854261] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:26.854268] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:26.854284] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:26.854295] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.854303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4247][T1_Occam][T1][YB427F000001-000635291C6C623F-0-0] [lt=7][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:26.854314] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:26.854325] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:26.854335] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:26.854344] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca3b0, start=1747302026853821, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:26.854356] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:26.854367] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f1abce563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:26.854384] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:26.854463] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1abce563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:26.854485] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:26.854504] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:26.854515] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:26.854531] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:26.860269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.870378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.880493] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.890603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.892555] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=20] ====== tenant freeze timer task ====== [2025-05-15 09:40:26.893576] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=32] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:26.895676] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:26.895676] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:26.898302] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:26.899051] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12329146870, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.899079] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.899101] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.899118] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.899129] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.899142] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1001}, delta=12329147927, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.899154] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.899168] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.899179] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:26.899188] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:26.899199] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.899209] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.899218] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026899030) [2025-05-15 09:40:26.900710] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.910820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.911101] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.911121] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:26.911142] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:26.911160] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:26.911169] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.911182] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.911191] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:26.911205] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302026911077, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.911218] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:26.911226] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:26.911285] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:26.911297] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:26.911307] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:26.911320] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:26.911313] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.920833] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:26.920930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.920865] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:2944224391}, scn={val:1747289697727835002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300530150701, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:1686016, ret_code_:0}}, next_to_submit_lsn_:{lsn:2944224391}, next_to_submit_scn_:{val:1747289697727835002, v:0}, base_lsn_:{lsn:2818400256}, base_scn_:{val:1747274404592930000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69205000, next_round_pread_size:3678208, curr_read_pos:3609237, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3609237, log_storage_:{start_lsn:{lsn:2940615154}, end_lsn:{lsn:2944224391}, read_buf:{buf_len_:3747840, buf_:0x7f1a69205000}, block_size:67104768, log_storage_:0x7f1ab0f4fbf0, log_storage_:{log_tail:{lsn:2944224980}, readable_log_tail:{lsn:2944224980}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:43, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:0, max_block_id:44, curr_writable_block_id:43}, logical_block_size_:67104768, curr_block_writable_size_:8384812, block_header_serialize_buf_:0x7f1ab0f508b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697727835001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697727835001, v:0}, data_checksum:2923900223, flag:13900}}, init_mode_version:0, accumulate_checksum:2881944827, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697727835001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:64, read_io_size:126980096, read_disk_cost_ts:1613509, start_lsn:{lsn:2818400256}}}}}}) [2025-05-15 09:40:26.920960] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=91] get wrs ts(ls_id={id:1}, delta=12329192361, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:26.920980] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.921004] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:26.921019] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.921033] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.921047] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.921058] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:26.921065] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026920822) [2025-05-15 09:40:26.921108] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:26.921132] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302026921102) [2025-05-15 09:40:26.921149] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302026721047, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:26.921175] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12329019868, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:26.921192] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.921211] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:26.921227] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.921238] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.921252] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.921262] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:26.921270] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026921165) [2025-05-15 09:40:26.925908] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:26.925946] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:26.928307] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.928332] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17343393, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.928349] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17343393, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.928376] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.928392] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.928404] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.928419] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.928435] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026928434, replica_locations:[]}) [2025-05-15 09:40:26.928453] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:26.928476] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:26.929744] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.929766] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.929781] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.929803] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.929816] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.929827] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.929841] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.929854] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026929854, replica_locations:[]}) [2025-05-15 09:40:26.929917] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=17341764, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=174, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:26.930503] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4799374245, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:26.930594] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:26.930609] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4799374245, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:26.931040] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.932846] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.932867] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.932882] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.932895] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.932913] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.932926] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.932936] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.932949] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.932960] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.932974] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:26.932985] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:26.932994] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:26.933015] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:26.933025] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.933037] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.933054] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.933065] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:26.933075] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:26.933086] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:26.933096] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302026933096, replica_locations:[]}) [2025-05-15 09:40:26.933114] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:26.933161] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:26.933175] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:26.933210] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=35] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:26.933230] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=20][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:26.933251] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=21] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=108, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:26.933273] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:26.933296] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=20][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:26.933305] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:26.941153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.951266] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.953824] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=47] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:26.953893] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] [COMPACTION DAG STATUS] (start_time=1747302016953051, end_time=1747302026953891, dag_status={}) [2025-05-15 09:40:26.953920] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.953959] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=39] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.953973] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.953981] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.953993] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.954001] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.954009] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.954039] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=30] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.954046] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:26.954061] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-15 09:40:26.954073] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-15 09:40:26.957272] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=38][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:54, tid:3961}]) [2025-05-15 09:40:26.957449] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:26.957476] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-05-15 09:40:26.957500] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:26.957517] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:26.957534] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11789847898, flush_io_finished_ret_:0, flush_io_finished_round_:1501, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:26.961377] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.971501] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.981614] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.987912] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:26.987944] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:26.987957] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:26.987983] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:26.987997] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:26.988010] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:26.988023] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:26.988036] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:26.988619] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:26.988640] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:26.988649] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:26.988657] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:26.988707] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:26.988717] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:26.988733] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:26.988747] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:26.988782] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:26.991726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:26.995088] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCC-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.995106] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCC-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:26.995129] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCC-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:26.995145] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCC-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:26.995157] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CCC-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:26.995169] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:26.995182] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.995193] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:26.995200] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302026995063, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:26.995213] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:26.995224] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:26.995298] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:26.996613] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCD-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.996631] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCD-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:26.996646] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCD-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:26.996680] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCD-0-0] [lt=33][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:26.996690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CCD-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:26.996701] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:26.996711] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:26.996720] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:26.996729] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302026996602, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:26.996737] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:26.996746] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:26.996786] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:26.998081] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:26.998357] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:26.999089] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:26.999117] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302026999082) [2025-05-15 09:40:26.999131] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302026799081, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:26.999166] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12329247161, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.999184] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.999205] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.999222] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:26.999230] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:26.999246] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12329247161, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:26.999263] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:26.999277] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:26.999287] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:26.999296] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:26.999309] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.999318] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:26.999327] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302026999145) [2025-05-15 09:40:26.999883] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:26.999917] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:26.999932] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:26.999943] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:26.999959] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.001840] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.010046] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.010063] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.010070] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.010199] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=52] tx gc loop thread is running(MTL_ID()=1002) [2025-05-15 09:40:27.010227] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] try gc retain ctx [2025-05-15 09:40:27.010235] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-05-15 09:40:27.010288] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=32] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289693216794002, v:0}, keep_alive_scn_:{val:1747289693216794002, v:0}, update_ts_:1747302027009986}) [2025-05-15 09:40:27.010889] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=22] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:27.010982] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=82] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:0, v:0}, keep_alive_scn_:{val:0, v:0}, update_ts_:0}) [2025-05-15 09:40:27.011191] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.011214] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.011234] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.011250] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.011274] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.011287] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.011299] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.011311] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.011322] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302027011167, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.011336] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.011344] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.011407] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.011418] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.011426] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.011435] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.011602] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:27.011952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.015523] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4606][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=30] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:27.020149] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.020173] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.020180] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.020892] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.020923] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027020884) [2025-05-15 09:40:27.020935] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302026820920, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.020971] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12329292689, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.020989] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.021010] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.021026] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.021039] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.021055] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.021066] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.021073] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027020956) [2025-05-15 09:40:27.021170] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:27.021199] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] get_min_unreplayed_log_info(lsn={lsn:5311387238}, scn={val:1747289697900328001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300529803179, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:5311387238}, next_to_submit_scn_:{val:1747289697900328001, v:0}, base_lsn_:{lsn:5167067136}, base_scn_:{val:1747274408663046000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a72405000, next_round_pread_size:3678208, curr_read_pos:2755105, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2755105, log_storage_:{start_lsn:{lsn:5308632133}, end_lsn:{lsn:5311387238}, read_buf:{buf_len_:3747840, buf_:0x7f1a72405000}, block_size:67104768, log_storage_:0x7f1a97bcbbf0, log_storage_:{log_tail:{lsn:5311387831}, readable_log_tail:{lsn:5311387831}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:79, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:133, min_block_id:61, max_block_id:80, curr_writable_block_id:79}, logical_block_size_:67104768, curr_block_writable_size_:56993609, block_header_serialize_buf_:0x7f1a97bcc8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697900328000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:431, scn_:{val:1747289697900328000, v:0}, data_checksum:4152786848, flag:25505}}, init_mode_version:0, accumulate_checksum:3961629555, curr_entry_is_padding:0, padding_entry_size:463, padding_entry_scn:{val:1747289697900328000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:76, read_io_size:145543168, read_disk_cost_ts:2159331, start_lsn:{lsn:5167067136}}}}}}) [2025-05-15 09:40:27.021285] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=85] get wrs ts(ls_id={id:1}, delta=12329120196, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.021301] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.021318] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:27.021333] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.021346] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.021360] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.021371] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.021381] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027021161) [2025-05-15 09:40:27.022066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.023022] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=6) [2025-05-15 09:40:27.028203] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1] LogLoopThread round_cost_time(us)(round_cost_time=25) [2025-05-15 09:40:27.030048] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.030075] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17241652, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.030091] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17241652, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.030105] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.030117] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=17241652, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.030139] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.030160] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.030175] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.030189] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.030200] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.030214] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.030229] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027030227, replica_locations:[]}) [2025-05-15 09:40:27.030249] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.030259] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.030273] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.030274] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.030283] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.030283] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:27.030320] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.030368] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11790920845, cache_obj->added_lc()=false, cache_obj->get_object_id()=15697, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.031663] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.031684] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.031699] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.031712] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.031720] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.031733] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.031748] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.031761] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.031774] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.031785] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.031799] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.031812] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027031811, replica_locations:[]}) [2025-05-15 09:40:27.031879] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=17239803, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=175, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.032177] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.040364] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.040389] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.040398] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.042292] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.044242] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.044248] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC9658-0-0] [lt=26][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302027044221}) [2025-05-15 09:40:27.044285] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:27.044301] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:27.044317] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.044695] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.044709] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.048432] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552613, support_ssl=false) [2025-05-15 09:40:27.048596] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=103] sm conn init succ(conn.sessid_=3221552613, sess.client_addr_="172.22.0.1:42168") [2025-05-15 09:40:27.048624] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552613, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.048804] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.048847] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.048878] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.048912] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=33][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.048931] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.048938] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.048960] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.048979] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552613, proxy_sessid=0) [2025-05-15 09:40:27.049015] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=16][errcode=0] server close connection(sessid=3221552613, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.049030] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552613, proxy_sessid=0) [2025-05-15 09:40:27.049037] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.049051] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9033-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552613, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.049108] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552613, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027047968, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.049134] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552613, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.049149] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552613, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027047968, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.049168] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552613) [2025-05-15 09:40:27.049188] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552613, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.050506] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.050532] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.050543] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.052399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.057501] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:443, tid:4580}]) [2025-05-15 09:40:27.060616] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.060653] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.060672] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.060682] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.060693] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.060701] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.060710] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:27.060720] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.062512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.067264] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:27.067293] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.067305] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:27.067313] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:27.067321] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:27.067400] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=78][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:27.070790] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.070810] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.070845] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.070874] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.070884] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.070893] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.070899] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:27.070906] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.072626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.080975] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.080993] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.081008] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.081018] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.081027] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.081035] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.081044] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:27.081053] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.082739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.083595] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552614, support_ssl=false) [2025-05-15 09:40:27.083597] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221552615, support_ssl=false) [2025-05-15 09:40:27.083692] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=35] sm conn init succ(conn.sessid_=3221552615, sess.client_addr_="172.22.0.1:42192") [2025-05-15 09:40:27.083697] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=33] sm conn init succ(conn.sessid_=3221552614, sess.client_addr_="172.22.0.1:42178") [2025-05-15 09:40:27.083712] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552615, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.083717] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221552614, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.083823] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552616, support_ssl=false) [2025-05-15 09:40:27.083872] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552616, sess.client_addr_="172.22.0.1:42208") [2025-05-15 09:40:27.083899] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552616, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.083913] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.083938] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.083947] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.083966] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.083977] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.084002] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.084015] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.084019] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.084010] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.084031] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.084038] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.084058] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.084055] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=42][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.084063] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=17] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084073] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.084074] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.084078] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552615, proxy_sessid=0) [2025-05-15 09:40:27.084087] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.084088] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.084099] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.084111] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=12][errcode=0] server close connection(sessid=3221552615, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084124] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552615, proxy_sessid=0) [2025-05-15 09:40:27.084117] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084120] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084132] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.084138] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552616, proxy_sessid=0) [2025-05-15 09:40:27.084139] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552614, proxy_sessid=0) [2025-05-15 09:40:27.084143] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9064-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552615, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.084168] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=16][errcode=0] server close connection(sessid=3221552616, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084182] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552616, proxy_sessid=0) [2025-05-15 09:40:27.084179] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=16][errcode=0] server close connection(sessid=3221552614, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084190] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:27.084195] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552614, proxy_sessid=0) [2025-05-15 09:40:27.084195] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552615, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.084201] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9034-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552616, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.084207] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552617, support_ssl=false) [2025-05-15 09:40:27.084216] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=20][errcode=-4016] session is null [2025-05-15 09:40:27.084219] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552615, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.084229] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552615, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.084228] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9025-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552614, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.084249] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552615) [2025-05-15 09:40:27.084254] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221552617, sess.client_addr_="172.22.0.1:42222") [2025-05-15 09:40:27.084245] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552616, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.084269] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552617, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.084269] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552615, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.084284] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221552614, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.084304] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221552616, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.084320] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] kill and revert session(conn.sessid_=3221552614, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.084327] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552616, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.084345] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552616) [2025-05-15 09:40:27.084358] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552616, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.084382] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221552614, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.084399] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552614) [2025-05-15 09:40:27.084410] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552614, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.084454] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.084497] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.084512] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.084529] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.084536] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.084547] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.084559] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=6] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084572] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552617, proxy_sessid=0) [2025-05-15 09:40:27.084591] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=8][errcode=0] server close connection(sessid=3221552617, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084602] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552617, proxy_sessid=0) [2025-05-15 09:40:27.084610] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.084623] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9065-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552617, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.084646] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221552618, support_ssl=false) [2025-05-15 09:40:27.084699] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552618, sess.client_addr_="172.22.0.1:42236") [2025-05-15 09:40:27.084715] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552618, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.084751] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552617, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.084777] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552617, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.084790] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552617, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.084807] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552617) [2025-05-15 09:40:27.084827] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552617, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.084868] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.084870] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552619, support_ssl=false) [2025-05-15 09:40:27.084893] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.084918] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.084941] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.084945] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552619, sess.client_addr_="172.22.0.1:42252") [2025-05-15 09:40:27.084951] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.084963] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.084968] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552619, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.084979] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.084990] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552618, proxy_sessid=0) [2025-05-15 09:40:27.085015] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=11][errcode=0] server close connection(sessid=3221552618, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.085030] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552618, proxy_sessid=0) [2025-05-15 09:40:27.085038] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.085051] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9035-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552618, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.085103] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552618, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.085126] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552618, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.085137] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552618, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027083883, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.085157] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552618) [2025-05-15 09:40:27.085155] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.085168] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552618, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.085188] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.085206] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.085227] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.085236] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.085249] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.085264] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.085280] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552619, proxy_sessid=0) [2025-05-15 09:40:27.085298] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=7][errcode=0] server close connection(sessid=3221552619, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.085312] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552619, proxy_sessid=0) [2025-05-15 09:40:27.085319] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.085320] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552620, support_ssl=false) [2025-05-15 09:40:27.085327] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9026-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552619, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.085369] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552620, sess.client_addr_="172.22.0.1:42254") [2025-05-15 09:40:27.085374] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552619, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302027084944, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.085386] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552620, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.085395] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552619, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.085407] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552619, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302027084944, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.085426] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552619) [2025-05-15 09:40:27.085439] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552619, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.085633] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.085690] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=35] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.085708] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.085724] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.085735] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.085742] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.085762] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.085778] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552620, proxy_sessid=0) [2025-05-15 09:40:27.085802] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=13][errcode=0] server close connection(sessid=3221552620, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.085814] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552620, proxy_sessid=0) [2025-05-15 09:40:27.085821] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.085832] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9066-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552620, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.085888] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552620, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027084944, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.085914] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552620, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.085923] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552620, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027084944, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.085940] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552620) [2025-05-15 09:40:27.085954] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552620, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.088396] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552621, support_ssl=false) [2025-05-15 09:40:27.088448] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221552621, sess.client_addr_="172.22.0.1:42264") [2025-05-15 09:40:27.088466] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552621, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.088684] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.088715] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.088731] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.088750] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.088762] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.088776] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.088791] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.088782] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552622, support_ssl=false) [2025-05-15 09:40:27.088790] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552623, support_ssl=false) [2025-05-15 09:40:27.088805] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552621, proxy_sessid=0) [2025-05-15 09:40:27.088826] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=8][errcode=0] server close connection(sessid=3221552621, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.088834] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=7][errcode=-4018] get session fail(ret=-4018, sessid=3221552621, proxy_sessid=0) [2025-05-15 09:40:27.088833] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552622, sess.client_addr_="172.22.0.1:42266") [2025-05-15 09:40:27.088842] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:27.088843] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552623, sess.client_addr_="172.22.0.1:42270") [2025-05-15 09:40:27.088850] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9036-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552621, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.088856] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552622, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.088860] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552623, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.088976] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552621, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027088124, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.089017] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36] kill and revert session(conn.sessid_=3221552621, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.089025] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552621, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027088124, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.089039] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552621) [2025-05-15 09:40:27.089056] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552621, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.089073] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.089097] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552624, support_ssl=false) [2025-05-15 09:40:27.089100] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.089087] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.089122] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.089141] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.089149] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552624, sess.client_addr_="172.22.0.1:42272") [2025-05-15 09:40:27.089158] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.089167] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.089164] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552624, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.089167] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=46] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.089182] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.089190] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.089198] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.089207] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.089208] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552622, proxy_sessid=0) [2025-05-15 09:40:27.089215] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.089232] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.089232] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=13][errcode=0] server close connection(sessid=3221552622, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.089247] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552622, proxy_sessid=0) [2025-05-15 09:40:27.089250] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552623, proxy_sessid=0) [2025-05-15 09:40:27.089255] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:27.089268] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=8][errcode=0] server close connection(sessid=3221552623, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.089269] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9027-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552622, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.089282] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552623, proxy_sessid=0) [2025-05-15 09:40:27.089290] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.089297] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9067-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552623, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.089321] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552622, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302027088124, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.089344] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552622, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.089354] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552622, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302027088124, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.089369] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552622) [2025-05-15 09:40:27.089358] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552623, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302027088124, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.089385] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552622, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.089400] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=37] kill and revert session(conn.sessid_=3221552623, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.089414] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.089419] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552623, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302027088124, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.089443] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552623) [2025-05-15 09:40:27.089450] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.089464] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552623, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.089501] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.089528] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.089539] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.089554] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.089582] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.089606] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552624, proxy_sessid=0) [2025-05-15 09:40:27.089651] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=16][errcode=0] server close connection(sessid=3221552624, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.089666] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552624, proxy_sessid=0) [2025-05-15 09:40:27.089675] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:27.089695] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9037-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552624, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.089748] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552624, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027089186, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.089771] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552624, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.089779] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552624, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027089186, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.089801] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552624) [2025-05-15 09:40:27.089812] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552624, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.091123] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.091146] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.091168] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.091188] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.091198] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.091211] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.091217] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:27.091230] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.092869] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.093848] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:27.098423] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.099163] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12329347509, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.099189] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.099216] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.099236] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.099245] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.099265] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12329347509, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.099283] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.099300] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.099311] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.099324] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.099334] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.099352] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.099368] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027099142) [2025-05-15 09:40:27.101298] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.101314] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.101330] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.101343] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.101351] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.101364] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.101370] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.101388] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.102671] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=21] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2025-05-15 09:40:27.102704] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2025-05-15 09:40:27.102716] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2025-05-15 09:40:27.102735] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2025-05-15 09:40:27.102753] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=15][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:27.102769] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=16][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:27.102783] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2025-05-15 09:40:27.102797] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:27.102812] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=14][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:27.102819] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=6][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:27.102833] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=12][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:27.102842] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=7][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:27.102857] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=13][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:27.102872] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:27.102893] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:27.102908] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:27.102920] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:27.102934] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:27.102944] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1, ret=-5019) [2025-05-15 09:40:27.102961] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:27.102976] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=13][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:27.102986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.103001] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=18][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:27.103026] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:27.103039] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:27.103045] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:27.103068] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:27.103084] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.103100] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C618F-0-0] [lt=15][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1"}, aret=-5019, ret=-5019) [2025-05-15 09:40:27.103109] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:27.103124] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:27.103133] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:27.103147] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302027102381, sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:27.103164] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:27.103173] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:27.103286] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:663) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1, tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-05-15 09:40:27.103305] WDIAG [STORAGE.DUP_TABLE] refresh_dup_ls_ (ob_dup_table_util.cpp:110) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] get duplicate ls status info failed(ret=-5019, tmp_dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-05-15 09:40:27.103330] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:202) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=24][errcode=0] refresh dup ls failed(tmp_ret=-5019, this={tenant_id_:1, dup_table_scan_timer_:0x7f1ac243dae8, dup_loop_worker_:0x7f1ac27066f0, min_dup_ls_status_info_:{tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, tenant_schema_dup_tablet_set_.size():0, scan_task_execute_interval_:10000000, last_dup_ls_refresh_time_:0, last_dup_schema_refresh_time_:0, last_scan_task_succ_time_:1747302015098096, max_execute_interval_:12003548}) [2025-05-15 09:40:27.111118] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.111140] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.111156] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.111172] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.111181] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.111196] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.111205] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.111219] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.111227] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302027111103, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.111242] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.111250] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.111297] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.111311] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.111320] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.111335] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.111471] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.111498] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.111514] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.111528] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.111537] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.111550] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.111556] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.113095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.114103] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:27.114125] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:27.114132] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=6] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:27.114154] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=21] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:27.116337] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.116364] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.116380] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:27.117372] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.117397] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.117409] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:27.117430] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:27.117444] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:27.117454] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:27.119627] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=17] gc stale ls task succ [2025-05-15 09:40:27.120970] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12329393089, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.120997] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.121026] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.121047] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.121057] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.121078] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.121092] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.121100] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027120945) [2025-05-15 09:40:27.121225] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.121263] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027121218) [2025-05-15 09:40:27.121283] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302026921165, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.121313] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12329220596, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.121332] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.121354] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:27.121371] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.121381] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.121399] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.121407] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.121420] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027121300) [2025-05-15 09:40:27.121631] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.121651] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.121669] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.121688] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.121699] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.121711] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.121718] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.123208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.126685] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:27.126714] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-05-15 09:40:27.126724] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:27.126745] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:27.126764] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11790017106, flush_io_finished_ret_:0, flush_io_finished_round_:1501, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:27.128287] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:27.128308] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.128320] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:27.131793] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.131814] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.131830] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.131844] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.131859] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.131866] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.131879] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.132011] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.132041] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17139690, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.132059] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17139690, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.132071] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.132086] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=17139690, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.132106] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.132126] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.132145] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.132163] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.132179] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.132198] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.132218] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027132216, replica_locations:[]}) [2025-05-15 09:40:27.132240] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.132262] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:175, local_retry_times:175, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:27.132285] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.132295] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:27.132314] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:27.132328] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:27.132335] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:27.132357] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:27.132373] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.132422] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11791022899, cache_obj->added_lc()=false, cache_obj->get_object_id()=15698, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.133319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.133593] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:27.133624] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=30][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.133660] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.133679] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.133696] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.133712] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.133720] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.133737] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.133756] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.133768] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.133784] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.133792] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.133810] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.133828] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027133827, replica_locations:[]}) [2025-05-15 09:40:27.133847] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:27.133864] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.133880] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:27.133900] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:27.133908] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:27.133923] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:27.133944] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:27.133961] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:27.133970] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:27.133984] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:27.133992] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:27.134005] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:27.134016] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:27.134031] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:27.134039] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:27.134059] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:27.134066] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:27.134081] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:27.134091] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:27.134110] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:27.134125] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:27.134140] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:27.134150] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:27.134167] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:27.134186] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=176, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:27.134216] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] will sleep(sleep_us=100000, remain_us=17137466, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=176, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.137796] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4289][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-05-15 09:40:27.137821] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4289][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:27.141954] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.141974] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.141994] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.142085] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=91][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.142094] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.142101] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.142136] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.143432] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.147958] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4291][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=22] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:27.152201] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.152218] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.152234] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.152248] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.152257] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.152270] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.152277] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.153543] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.156795] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2025-05-15 09:40:27.156824] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2025-05-15 09:40:27.156835] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2025-05-15 09:40:27.156847] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=11][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2025-05-15 09:40:27.156864] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:27.156873] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=8][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:27.156893] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=15][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2025-05-15 09:40:27.156911] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=16][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:27.156921] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:27.156934] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=13][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:27.156943] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:27.157005] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=61][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:27.157012] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:27.157025] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=12][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:27.157041] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.157080] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.157162] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61A9-0-0] [lt=0] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:27.158477] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2025-05-15 09:40:27.158541] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61A9-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.158630] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61A9-0-0] [lt=0] cal_all_part_disk_default_percentage succ(data_dir="/root/ob/observer/store/sstable", clog_dir="/root/ob/observer/store/clog", shared_mode=true, data_disk_total_size=3736332664832, data_disk_default_percentage=60, clog_disk_total_size=3736332664832, clog_disk_default_percentage=30) [2025-05-15 09:40:27.158652] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61A9-0-0] [lt=21] decide disk size finished(suggested_disk_size=21474836480, suggested_disk_percentage=0, default_disk_percentage=30, total_space=3736332664832, disk_size=21474836480) [2025-05-15 09:40:27.158669] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61A9-0-0] [lt=17] get_log_disk_info_in_config(suggested_data_disk_size=21474836480, suggested_clog_disk_size=21474836480, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=21474836480, log_disk_percentage=0, total_log_disk_size=3736332664832) [2025-05-15 09:40:27.158689] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61A9-0-0] [lt=16] try_resize success(ret=0, log_disk_size=21474836480, total_log_disk_size=3736332664832, this={dir::"/root/ob/observer/store/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:21474836480, next_total_size:21474836480, status:0}, min_block_id:154, max_block_id:381, min_log_disk_size_for_all_tenants_:21474836480, is_inited:true}) [2025-05-15 09:40:27.158711] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61A9-0-0] [lt=23] refresh tenant config(tenants=[], ret=-5019) [2025-05-15 09:40:27.162349] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.162371] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.162392] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.162411] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.162420] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.162433] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.162440] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.163657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.168142] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6469-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:27.168211] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6469-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.172514] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.172532] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.172547] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.172561] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.172570] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.172583] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.172589] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.173773] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.173899] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4501][T1001_ArcSender][T1001][YB427F000001-000635291CBC6112-0-0] [lt=18][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:27.174113] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.174331] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.174859] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.174921] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.182661] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.182681] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.182700] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.182718] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.182727] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.182740] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.182747] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.184235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.190576] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:27.192817] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.192838] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.192854] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.192868] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.192876] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.192889] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.192896] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.194892] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.198484] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.198552] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=9][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:27.198574] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.198586] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=8][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.198674] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=89][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.198686] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.198702] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.198720] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.198739] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.198757] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.198775] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.198785] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.198804] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.198835] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=14] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.198852] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=14] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:27.199206] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001) [2025-05-15 09:40:27.199227] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.199250] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027199202) [2025-05-15 09:40:27.199270] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302026999144, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.199296] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:27.199326] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get_min_unreplayed_log_info(lsn={lsn:934287681}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302027196376, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:934287681}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:872361984}, base_scn_:{val:1747274406063927000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69605000, next_round_pread_size:3678208, curr_read_pos:3075493, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3075493, log_storage_:{start_lsn:{lsn:931212188}, end_lsn:{lsn:934287681}, read_buf:{buf_len_:3747840, buf_:0x7f1a69605000}, block_size:67104768, log_storage_:0x7f1a73fd1bf0, log_storage_:{log_tail:{lsn:934288498}, readable_log_tail:{lsn:934288498}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:13, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:143, min_block_id:0, max_block_id:14, curr_writable_block_id:13}, logical_block_size_:67104768, curr_block_writable_size_:5178254, block_header_serialize_buf_:0x7f1a73fd28b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:2923900223, flag:16486}}, init_mode_version:0, accumulate_checksum:910368143, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}}}}) [2025-05-15 09:40:27.199422] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=92] get wrs ts(ls_id={id:1}, delta=12329447986, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.199442] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.199466] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.199485] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.199508] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.199521] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12329447986, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.199537] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.199555] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.199571] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.199579] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.199595] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.199608] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.199616] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027199286) [2025-05-15 09:40:27.200527] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=15] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:27.200617] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BB-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.202968] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.202984] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.203003] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.203022] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.203032] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.203045] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.203052] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.203068] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.205009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.211239] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.211263] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.211278] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.211298] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.211307] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.211324] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.211333] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.211347] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.211356] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302027211222, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.211372] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.211381] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.211435] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.211450] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.211458] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.211472] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.213145] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.213164] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.213182] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.213200] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.213226] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.213244] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.213251] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.215137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.216408] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.216435] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.216450] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:27.217434] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.217457] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.217470] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:27.217502] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:27.217517] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:27.217527] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:27.221008] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.221049] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027221001) [2025-05-15 09:40:27.221070] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302027020948, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.221105] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12329492491, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.221127] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.221156] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.221181] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.221291] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12329319998, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.221313] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.221335] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.223126] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=32] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:27.223325] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.223346] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.223360] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.223375] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.223383] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.223396] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.225283] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.228344] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:27.228370] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.228382] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:27.233467] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.233485] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.233515] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.233644] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=130][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.233654] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.233667] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.234341] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.234366] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17037355, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.234386] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=17037355, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.234397] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.234412] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=17037355, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.234431] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.234450] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.234468] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.234486] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.234512] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.234532] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.234552] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027234549, replica_locations:[]}) [2025-05-15 09:40:27.234573] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.234595] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:176, local_retry_times:176, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:27.234619] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.234634] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:27.234648] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:27.234661] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:27.234667] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=5][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:27.234687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:27.234703] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.234741] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11791125219, cache_obj->added_lc()=false, cache_obj->get_object_id()=15699, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.235396] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.235916] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:27.235952] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=35][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.235988] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.236005] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.236022] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.236035] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.236049] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.236065] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.236083] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.236099] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.236116] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.236126] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.236143] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.236155] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027236154, replica_locations:[]}) [2025-05-15 09:40:27.236175] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:27.236191] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.236207] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:27.236226] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:27.236234] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:27.236249] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:27.236275] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:27.236293] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:27.236301] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:27.236316] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:27.236324] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:27.236337] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:27.236349] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:27.236364] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:27.236372] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:27.236386] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:27.236395] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:27.236408] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:27.236417] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:27.236434] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:27.236448] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:27.236457] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:27.236471] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:27.236481] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:27.236503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=177, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:27.236528] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=17035154, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=177, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.243738] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.243759] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.243776] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.243789] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.243799] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.243812] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.245510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.251844] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.251867] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:27.251877] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:27.251897] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:27.251909] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:27.251939] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=1] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:27.253879] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.253898] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.253915] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.253933] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.253942] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.253955] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.255623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.257092] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:27.257120] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-05-15 09:40:27.257135] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:27.257151] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:27.257174] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11790147517, flush_io_finished_ret_:0, flush_io_finished_round_:1501, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:27.262410] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:27.264041] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.264068] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.264089] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.264105] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.264120] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.264128] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.265745] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.270313] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6451-0-0] [lt=16][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.270335] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6451-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.270352] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6451-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.270370] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6451-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.270380] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6451-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.270396] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:27.270413] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.270421] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.270434] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302027270295, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:27.270451] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.270461] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:27.270525] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1001, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:27.270574] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:27.271771] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6452-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.271789] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6452-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.271801] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6452-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.271817] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6452-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.271824] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6452-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.271838] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:27.271849] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.271862] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.271870] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302027271763, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:27.271884] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.271891] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:27.271931] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] fail to select for update snapshot_gc_scn(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001) [2025-05-15 09:40:27.271947] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to load updated info(ret=-5627) [2025-05-15 09:40:27.271955] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to try update info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:27.274202] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.274220] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.274239] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.274257] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.274274] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.274281] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.275877] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.284355] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.284373] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.284388] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.284401] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.284409] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.284421] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.285781] WDIAG load_file_to_string (utility.h:673) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:27.285806] WDIAG get_ethernet_speed (utility.cpp:575) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:27.285816] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:27.285827] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:27.285989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.294120] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=9] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:27.294495] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.294517] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.294536] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.294555] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.294564] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.294577] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.296101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.298545] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.299289] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12329547295, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.299317] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.299346] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.299373] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1001}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:27.299400] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get_min_unreplayed_log_info(lsn={lsn:747348431}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302027296663, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:747348431}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:738152448}, base_scn_:{val:1747289697751194001, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a5a205000, next_round_pread_size:3678208, curr_read_pos:1839725, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1839725, log_storage_:{start_lsn:{lsn:745508706}, end_lsn:{lsn:747348431}, read_buf:{buf_len_:3747840, buf_:0x7f1a5a205000}, block_size:67104768, log_storage_:0x7f1a73f61bf0, log_storage_:{log_tail:{lsn:747348553}, readable_log_tail:{lsn:747348553}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:11, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:140, min_block_id:0, max_block_id:12, curr_writable_block_id:11}, logical_block_size_:67104768, curr_block_writable_size_:57908663, block_header_serialize_buf_:0x7f1a73f628b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:154833137, flag:60041}}, init_mode_version:0, accumulate_checksum:2356512705, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}}}}) [2025-05-15 09:40:27.299503] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=103] get wrs ts(ls_id={id:1001}, delta=12329547295, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.299514] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.301557] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=43][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:27.301574] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.301584] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:27.301607] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:27.304644] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.304660] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.304718] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.304731] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.304739] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.304751] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.304765] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.306214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.311378] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.311402] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.311423] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.311446] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.311455] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.311471] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.311498] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=26][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.311507] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.311571] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=62][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302027311348, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.311588] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.311597] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.311674] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.311690] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.311698] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.311713] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.314849] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:27.314868] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:27.314883] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.314897] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:27.314905] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.314913] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.316328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.316467] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.317501] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.321086] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=12329592776, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.321114] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.321342] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.321376] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027321334) [2025-05-15 09:40:27.321398] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302027121300, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.321436] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12329420283, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.321455] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.325002] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.325025] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.326441] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.328402] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.335103] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.335120] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.336682] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.336709] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16935026, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.336731] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16935026, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.336749] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.336760] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=16935026, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.336778] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.336800] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.336822] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.336841] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.336850] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.336870] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.336894] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027336891, replica_locations:[]}) [2025-05-15 09:40:27.336917] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.336948] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.336964] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:27.337009] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.337072] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11791227548, cache_obj->added_lc()=false, cache_obj->get_object_id()=15700, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.336559] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1746497860185946, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:40, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:148316119, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300520045442, tenant_role:1}) [2025-05-15 09:40:27.337093] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=540] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:5, max_cpu:5, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:5, }}, mode:0, create_timestamp:1747300516548130, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"5.000000000000000000e+00", unit_max_cpu:"5.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:52, max_worker_cnt:150, stopped:0, worker_us:17954240, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:52, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300516557108, tenant_role:0}) [2025-05-15 09:40:27.337600] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=512] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:2.5, max_cpu:2.5, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:2, }}, mode:0, create_timestamp:1747300516541652, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"2.500000000000000000e+00", unit_max_cpu:"2.500000000000000000e+00", total_worker_cnt:35, min_worker_cnt:27, max_worker_cnt:150, stopped:0, worker_us:11483527, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:27, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300516564847, tenant_role:0}) [2025-05-15 09:40:27.338543] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.338567] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.338074] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=479] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"singi_unit", resource:{min_cpu:1.3, max_cpu:1.3, memory_size:"4.02266846GB", log_disk_size:"1.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:13, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:13, }}, mode:0, create_timestamp:1746497897813453, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.300000000000000044e+00", unit_max_cpu:"1.300000000000000044e+00", total_worker_cnt:15, min_worker_cnt:7, max_worker_cnt:150, stopped:0, worker_us:46095038, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:7, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300522736765, tenant_role:1}) [2025-05-15 09:40:27.338586] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.338602] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.338616] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.338632] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.338651] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.338669] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.338686] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.338694] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.338707] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.338726] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027338725, replica_locations:[]}) [2025-05-15 09:40:27.338800] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16932882, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=178, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.338596] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=527] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"singi_unit", resource:{min_cpu:13, max_cpu:13, memory_size:"36.2040161GB", log_disk_size:"13.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:13, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:13, }}, mode:0, create_timestamp:1746497897813453, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=8, log_id=1304693, offset=56638805}, ls_meta_entry:{[ver=1,mode=0,seq=619756][2nd=8512]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.300000000000000000e+01", unit_max_cpu:"1.300000000000000000e+01", total_worker_cnt:62, min_worker_cnt:54, max_worker_cnt:463, stopped:0, worker_us:216249632, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:54, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300525931346, tenant_role:0}) [2025-05-15 09:40:27.339313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=713][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.345205] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.345229] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.349461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.351432] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:27.351468] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.351484] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:27.351511] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:27.351519] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:27.351616] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:27.351629] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.351642] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:27.351648] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:27.351655] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:27.351714] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:27.352275] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:27.352348] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:27.352419] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:27.352503] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:27.352535] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:8399, total:8409) [2025-05-15 09:40:27.352575] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:27.352635] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=57] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:8399, total:8409) [2025-05-15 09:40:27.352668] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:27.352693] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:27.352710] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:27.352728] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628762][2nd=1889]}) [2025-05-15 09:40:27.355306] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.355323] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.359586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.365404] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.365428] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.370734] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.375508] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.375526] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.380852] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.385603] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.385625] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.390968] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.395703] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.395724] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.395992] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=19] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.395996] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.396028] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.396085] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.396271] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.396673] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.396767] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:27.398601] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.399329] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.399362] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027399323) [2025-05-15 09:40:27.399401] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302027199285, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.399444] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12329647284, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.399467] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.399514] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12329647284, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.399531] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.401086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.405805] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.405823] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.405837] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.411201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.411421] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.411421] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.411450] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=28][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.411471] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.411504] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=30][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.411515] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.411602] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=87][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.411612] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.411626] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.411634] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302027411395, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.411651] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.411661] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.411729] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.411743] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.411751] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.411764] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.415927] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.415945] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.421125] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.421153] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027421119) [2025-05-15 09:40:27.421167] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302027221082, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.421209] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=12329692750, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.421228] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.421316] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.421408] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12329520257, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.421431] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.423181] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:27.426022] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.426043] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.431429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.432643] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:27.432683] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=1] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:27.436117] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.436134] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.438927] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.438951] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16832771, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.438968] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16832771, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.438985] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.438994] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=16832771, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.439012] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.439032] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.439050] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.439068] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.439077] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.439090] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.439110] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027439109, replica_locations:[]}) [2025-05-15 09:40:27.439131] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.439157] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.439172] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:27.439199] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.439249] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11791329726, cache_obj->added_lc()=false, cache_obj->get_object_id()=15701, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.440315] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.440347] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.440365] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.440382] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.440389] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.440405] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.440423] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.440435] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.440453] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.440462] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.440480] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.440508] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027440507, replica_locations:[]}) [2025-05-15 09:40:27.440570] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16831111, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=179, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.441541] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.446216] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.446240] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.451652] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.456316] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.456333] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.458299] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:77, tid:4630}, {errcode:-5019, dropped:136, tid:4776}]) [2025-05-15 09:40:27.461767] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.466412] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.466435] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.466443] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.471884] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.476527] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.476546] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.476555] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.478459] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [4578][T1002_TimerWK1_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=14] BlockGCTimerTask success(ret=0, cost_time_us=6, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"127.0.0.1:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):195, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-15 09:40:27.481998] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.486637] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.486658] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.486666] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.492114] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.494510] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=35] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:27.495384] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4578][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=48] try inc thread count(*this={name:TimerWK, this:0x7f1a8f8b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:6036113994, tenant_id:1002}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:27.495396] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.495423] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.495440] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.495451] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.495470] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:27.495484] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:27.495515] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:27.495528] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:27.495561] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:27.495634] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [4577][T1002_TimerWK0_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=19] clear dtl interm result cost(us)(clear_cost=10393, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-15 09:40:27.495777] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4578][T1002_TimerWK1][T1002][Y0-0000000000000000-0-0] [lt=26] inc thread count(*this={name:TimerWK, this:0x7f1a8f8b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:3, threads_idle_time:6036114401, tenant_id:1002}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:27.496749] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.496772] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.496780] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.497241] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C65FC-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.497264] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C65FC-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.497291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C65FC-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.497311] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C65FC-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.497321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FC-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.497338] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:27.497349] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.497365] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.497376] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302027497205, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:27.497393] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.497381] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=12][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.497402] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.497402] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.497413] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.497429] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=16][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.497439] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=9][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:27.497466] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=14][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:27.497486] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.497511] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C665A-0-0] [lt=23][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5157, ret=-5157) [2025-05-15 09:40:27.497521] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:27.497537] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.497547] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:27.497562] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78cb0f0, start=1747302027497349, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:27.497579] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:27.497727] INFO [STORAGE] runTimerTask (ob_tenant_restore_info_mgr.cpp:281) [4578][T1002_TimerWK1_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=0] refresh restore info [2025-05-15 09:40:27.498662] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.498685] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.498707] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=21][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.498716] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.498730] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.498738] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=7][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:27.498756] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=13][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:27.498772] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.498780] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C65FD-0-0] [lt=7][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5157, ret=-5157) [2025-05-15 09:40:27.498795] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:27.498803] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.498817] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:27.498824] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78ca3b0, start=1747302027498675, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:27.498833] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:27.498847] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] execute sql failed(ret=-5157, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:27.498885] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to select for update snapshot_gc_scn(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.498901] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5157] failed to load updated info(ret=-5157) [2025-05-15 09:40:27.498909] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to try update info(tmp_ret=-5157, tmp_ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:27.499412] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12329747439, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.499440] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.499469] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.499502] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.499514] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.499535] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12329747439, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.499552] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.499570] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.499580] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.499593] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.499604] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.499618] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.499626] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027499384) [2025-05-15 09:40:27.502231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.506867] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.506890] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.506898] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.506909] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.511460] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.512346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.516999] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.517026] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.517104] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.517629] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302027516595, register_time_us=-1, curr_time_us=1747302027517627) [2025-05-15 09:40:27.517660] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:27.517685] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1] re_register_parent reason(palf_id=1001, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302027517652, register_time_us=-1, curr_time_us=1747302027517685) [2025-05-15 09:40:27.517702] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1001, self="127.0.0.1:2882") [2025-05-15 09:40:27.521202] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12329792986, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.521229] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.521255] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.521276] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.521287] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.521306] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.521344] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.521352] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027521186) [2025-05-15 09:40:27.521460] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.521497] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027521454) [2025-05-15 09:40:27.521518] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302027321415, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.521539] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12329620493, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.521557] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.521578] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:27.521595] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.521603] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.521635] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.521649] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.521656] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027521528) [2025-05-15 09:40:27.522459] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.527181] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.527199] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.527206] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.534777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.535604] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:27.537285] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.537307] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.537315] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.540720] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.540748] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16730988, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.540769] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16730988, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.540788] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.540798] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=16730988, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.540817] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.540839] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.540885] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=40][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.540904] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.540914] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.540935] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.540958] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027540955, replica_locations:[]}) [2025-05-15 09:40:27.540981] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.541010] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.541024] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:27.541064] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.541121] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11791431597, cache_obj->added_lc()=false, cache_obj->get_object_id()=15702, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.541339] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:27.542556] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.542579] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.542597] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.542608] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.542622] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.542639] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.542658] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.542675] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.542692] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.542702] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.542720] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.542744] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027542743, replica_locations:[]}) [2025-05-15 09:40:27.542817] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16728864, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=180, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.544896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.545134] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.545179] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.545204] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:27.545223] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:27.545242] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:27.545261] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.545631] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.545647] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.547398] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.547422] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.547430] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.549842] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552638, support_ssl=false) [2025-05-15 09:40:27.549975] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221552638, sess.client_addr_="172.22.0.1:42280") [2025-05-15 09:40:27.549996] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552638, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.550234] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.550276] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.550318] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.550356] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=35][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.550375] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.550384] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.550409] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.550433] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=20][errcode=-4018] get session fail(ret=-4018, sessid=3221552638, proxy_sessid=0) [2025-05-15 09:40:27.550472] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=17][errcode=0] server close connection(sessid=3221552638, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.550500] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=28][errcode=-4018] get session fail(ret=-4018, sessid=3221552638, proxy_sessid=0) [2025-05-15 09:40:27.550515] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:27.550525] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9028-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552638, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.550591] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552638, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027549255, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.550622] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221552638, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.550642] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552638, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027549255, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.550663] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552638) [2025-05-15 09:40:27.550681] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552638, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.555020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.556346] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC668D-0-0] [lt=32][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.556383] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC668D-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.557509] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.557527] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.557536] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.565171] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.567617] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.567640] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.567649] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.575289] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.577735] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.577752] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.577760] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.585174] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552640, support_ssl=false) [2025-05-15 09:40:27.585208] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552641, support_ssl=false) [2025-05-15 09:40:27.585254] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=45] sm conn init succ(conn.sessid_=3221552640, sess.client_addr_="172.22.0.1:42282") [2025-05-15 09:40:27.585259] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552641, sess.client_addr_="172.22.0.1:42286") [2025-05-15 09:40:27.585273] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552641, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.585272] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552640, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.585301] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552642, support_ssl=false) [2025-05-15 09:40:27.585327] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221552642, sess.client_addr_="172.22.0.1:42284") [2025-05-15 09:40:27.585339] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552642, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.585405] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.585425] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.585425] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.585463] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.585467] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.585511] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.585510] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.585510] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.585531] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.585529] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.585541] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.585537] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.585554] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.585558] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.585554] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.585571] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.585574] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.585587] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.585581] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.585595] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.585599] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552640, proxy_sessid=0) [2025-05-15 09:40:27.585607] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.585621] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.585629] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552641, proxy_sessid=0) [2025-05-15 09:40:27.585631] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=12][errcode=0] server close connection(sessid=3221552640, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.585641] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552642, proxy_sessid=0) [2025-05-15 09:40:27.585646] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552640, proxy_sessid=0) [2025-05-15 09:40:27.585656] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:27.585675] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9038-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552640, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.585662] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=58] new mysql sessid created(conn.sessid_=3221552643, support_ssl=false) [2025-05-15 09:40:27.585717] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=10][errcode=0] server close connection(sessid=3221552642, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.585717] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=17][errcode=0] server close connection(sessid=3221552641, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.585730] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552641, proxy_sessid=0) [2025-05-15 09:40:27.585728] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552642, proxy_sessid=0) [2025-05-15 09:40:27.585725] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552640, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.585740] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:27.585739] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:27.585748] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552640, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.585749] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9029-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552642, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.585756] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=57] sm conn init succ(conn.sessid_=3221552643, sess.client_addr_="172.22.0.1:42296") [2025-05-15 09:40:27.585759] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552640, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.585754] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9068-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552641, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.585778] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552640) [2025-05-15 09:40:27.585782] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552643, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.585794] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552640, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.585814] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552641, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.585831] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552642, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.585838] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552641, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.585849] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552642, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.585847] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552641, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.585862] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552641) [2025-05-15 09:40:27.585870] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552642, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.585886] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552642) [2025-05-15 09:40:27.585878] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552641, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.585897] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552642, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.586003] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.586077] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=29] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.586112] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.586145] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=29][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.586159] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.586169] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.586197] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.586216] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552643, proxy_sessid=0) [2025-05-15 09:40:27.586260] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=15][errcode=0] server close connection(sessid=3221552643, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.586282] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=23][errcode=-4018] get session fail(ret=-4018, sessid=3221552643, proxy_sessid=0) [2025-05-15 09:40:27.586341] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=57][errcode=-4016] session is null [2025-05-15 09:40:27.586390] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9039-0-0] [lt=47] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552643, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.586485] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552643, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.586599] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=111] kill and revert session(conn.sessid_=3221552643, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.586616] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552643, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302027585199, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.586659] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=41] mark session id unused(conn.sessid_=3221552643) [2025-05-15 09:40:27.586699] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] connection close(sessid=3221552643, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.586769] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=46] new mysql sessid created(conn.sessid_=3221552644, support_ssl=false) [2025-05-15 09:40:27.586814] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552644, sess.client_addr_="172.22.0.1:42306") [2025-05-15 09:40:27.586833] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552644, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.586870] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552645, support_ssl=false) [2025-05-15 09:40:27.586914] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552645, sess.client_addr_="172.22.0.1:42316") [2025-05-15 09:40:27.586936] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552645, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.587009] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.587022] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] new mysql sessid created(conn.sessid_=3221552646, support_ssl=false) [2025-05-15 09:40:27.587043] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.587059] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.587078] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.587086] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.587085] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221552646, sess.client_addr_="172.22.0.1:42320") [2025-05-15 09:40:27.587093] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.587112] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.587101] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552646, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.587128] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552644, proxy_sessid=0) [2025-05-15 09:40:27.587128] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.587146] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=8][errcode=0] server close connection(sessid=3221552644, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.587148] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.587163] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552644, proxy_sessid=0) [2025-05-15 09:40:27.587166] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.587172] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:27.587184] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.587192] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.587185] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902A-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552644, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.587207] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.587220] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.587234] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552645, proxy_sessid=0) [2025-05-15 09:40:27.587235] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552644, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027586260, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.587252] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.587258] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=12][errcode=0] server close connection(sessid=3221552645, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.587269] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552645, proxy_sessid=0) [2025-05-15 09:40:27.587267] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221552644, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.587276] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:27.587283] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9069-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552645, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.587294] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.587279] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552644, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027586260, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.587320] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=53][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.587313] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31] mark session id unused(conn.sessid_=3221552644) [2025-05-15 09:40:27.587345] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.587368] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=22][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.587353] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] connection close(sessid=3221552644, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.587376] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.587397] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.587415] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552646, proxy_sessid=0) [2025-05-15 09:40:27.587402] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552645, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302027586260, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.587428] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552645, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.587439] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=9][errcode=0] server close connection(sessid=3221552646, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.587451] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552646, proxy_sessid=0) [2025-05-15 09:40:27.587458] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.587466] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903A-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552646, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.587507] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552645, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302027586260, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.587538] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] mark session id unused(conn.sessid_=3221552645) [2025-05-15 09:40:27.587550] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552646, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302027586260, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.587558] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552645, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.587605] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=50] kill and revert session(conn.sessid_=3221552646, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.587630] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552646, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302027586260, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.587646] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552646) [2025-05-15 09:40:27.587666] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552646, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.587805] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.587835] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.587860] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.590044] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552647, support_ssl=false) [2025-05-15 09:40:27.590097] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221552647, sess.client_addr_="172.22.0.1:42328") [2025-05-15 09:40:27.590115] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552647, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.590260] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.590296] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.590315] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.590335] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.590343] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.590356] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.590369] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.590386] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552647, proxy_sessid=0) [2025-05-15 09:40:27.590403] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=8][errcode=0] server close connection(sessid=3221552647, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.590418] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552647, proxy_sessid=0) [2025-05-15 09:40:27.590424] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.590438] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902B-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552647, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.590493] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552647, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027589433, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.590506] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552648, support_ssl=false) [2025-05-15 09:40:27.590519] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552647, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.590527] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552647, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027589433, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.590547] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552647) [2025-05-15 09:40:27.590559] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221552648, sess.client_addr_="172.22.0.1:42344") [2025-05-15 09:40:27.590562] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552647, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.590575] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552648, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.590602] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552649, support_ssl=false) [2025-05-15 09:40:27.590636] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552649, sess.client_addr_="172.22.0.1:42356") [2025-05-15 09:40:27.590652] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552649, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.590796] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.590841] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=22] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.590856] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.590859] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.590871] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221552650, support_ssl=false) [2025-05-15 09:40:27.590879] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.590878] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.590891] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.590898] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.590911] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.590902] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=37][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.590914] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552650, sess.client_addr_="172.22.0.1:42370") [2025-05-15 09:40:27.590926] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552648, proxy_sessid=0) [2025-05-15 09:40:27.590930] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.590931] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552650, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.590947] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=9][errcode=0] server close connection(sessid=3221552648, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.590948] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.590961] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552648, proxy_sessid=0) [2025-05-15 09:40:27.590962] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.590969] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:27.590978] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.590980] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906A-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552648, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.590997] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552649, proxy_sessid=0) [2025-05-15 09:40:27.591022] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=8][errcode=0] server close connection(sessid=3221552649, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.591037] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552649, proxy_sessid=0) [2025-05-15 09:40:27.591030] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552648, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027590494, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.591045] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:27.591053] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552648, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.591057] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903B-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552649, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.591064] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552648, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302027590494, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.591081] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552648) [2025-05-15 09:40:27.591093] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552648, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.591127] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:27.591121] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552649, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027590494, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.591151] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:27.591166] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:27.591151] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552649, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.591183] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:27.591190] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:27.591185] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552649, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302027590494, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.591198] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:27.591215] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.591212] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] mark session id unused(conn.sessid_=3221552649) [2025-05-15 09:40:27.591225] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552650, proxy_sessid=0) [2025-05-15 09:40:27.591238] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221552649, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.591265] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=31][errcode=0] server close connection(sessid=3221552650, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.591279] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552650, proxy_sessid=0) [2025-05-15 09:40:27.591286] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:27.591292] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902C-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552650, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:27.591337] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552650, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027590494, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:27.591361] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552650, proxy_sessid=0, ret=0) [2025-05-15 09:40:27.591367] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552650, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302027590494, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:27.591385] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552650) [2025-05-15 09:40:27.591398] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552650, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:27.595522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.597940] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.597957] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.597965] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.598721] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.599449] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.599474] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027599443) [2025-05-15 09:40:27.599508] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302027399417, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.599541] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12329847762, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.599561] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.599585] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.599602] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.599613] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.599630] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12329847762, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.599649] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.599666] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.599678] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.599685] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.599699] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.599710] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.599721] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027599523) [2025-05-15 09:40:27.604117] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=26] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4801404911, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:27.604358] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=29] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4801404917, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:27.605638] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.608048] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.608070] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.608077] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.608091] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.611640] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.615752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.618171] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.618191] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.618198] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.621246] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.621278] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027621238) [2025-05-15 09:40:27.621294] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302027421194, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.621331] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12329893329, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.621350] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.621374] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.621393] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.621403] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.621418] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.621427] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.621437] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027621311) [2025-05-15 09:40:27.621529] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12329720836, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.621549] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.621567] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:27.621580] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.621589] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.621602] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.621611] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.621620] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027621514) [2025-05-15 09:40:27.623272] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=157] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:27.625865] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.628273] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.628290] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.628297] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.628412] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EAB14-0-0] [lt=23][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302027198830}) [2025-05-15 09:40:27.635977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.638377] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.638398] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.638405] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.642954] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.642978] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16628748, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.643011] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=33][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16628748, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.643026] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:27.643039] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=16628748, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:27.643054] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:27.643071] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:27.643087] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.643102] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.643115] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.643131] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.643148] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027643146, replica_locations:[]}) [2025-05-15 09:40:27.643166] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.643191] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.643202] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:27.643230] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.643281] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11791533757, cache_obj->added_lc()=false, cache_obj->get_object_id()=15703, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:27.644553] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.644573] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.644589] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.644611] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.644625] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.644637] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.644653] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.644663] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027644663, replica_locations:[]}) [2025-05-15 09:40:27.644739] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16626943, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=181, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.647565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.648484] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.648509] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.648516] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.657680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.658597] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.658621] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.658633] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.658739] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:48, tid:4602}]) [2025-05-15 09:40:27.667797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.668780] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.668796] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.668804] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.677910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.678888] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.678904] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.678910] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.688025] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.688987] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.689005] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.689013] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.694774] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44868141875, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:27.698139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.698779] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.699089] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.699103] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.699111] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.699519] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12329948149, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.699544] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.699576] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.699605] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.699630] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.699657] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=12329948149, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.699671] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.699686] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.699698] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.699706] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.699719] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.699729] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.699740] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027699501) [2025-05-15 09:40:27.708253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.709186] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.709201] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.709209] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.709220] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.711623] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.711647] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.711670] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.711696] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.711709] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.711721] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.711732] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.711743] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302027711600, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.711756] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.711766] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.711833] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.711844] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.711853] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.711862] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.718368] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.719299] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.719314] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.719321] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.721315] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12329992609, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.721340] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.721362] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.721379] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.721389] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.721403] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.721413] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.721421] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027721296) [2025-05-15 09:40:27.721578] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.721613] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027721572) [2025-05-15 09:40:27.721630] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302027521528, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.721653] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12329821172, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.721667] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.721683] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:27.721698] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.721707] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.721720] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.721729] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.721737] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027721642) [2025-05-15 09:40:27.728486] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.729398] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.729413] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.729420] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.730881] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x7f1a8f8b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:6037172227, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:27.730910] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:27.730924] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try_wait return eagain(this=0x7f1a8f8b0e70, thread=0x7f1a50617e40) [2025-05-15 09:40:27.730937] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] inc thread count(*this={name:TimerWK, this:0x7f1a8f8b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:6037172227, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:27.738601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.739499] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.739514] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.739521] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.740867] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3953][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=23] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:27.744866] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.744888] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16526832, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.744904] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16526832, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.744929] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.744945] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.744958] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.744971] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.744987] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027744985, replica_locations:[]}) [2025-05-15 09:40:27.745004] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.745028] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.746197] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.746218] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.746234] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.746255] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.746270] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.746282] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.746298] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.746312] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027746311, replica_locations:[]}) [2025-05-15 09:40:27.746372] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16525310, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=182, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.748708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.749604] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.749620] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.749627] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.752000] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=61][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.752023] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:27.752040] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:27.752059] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:27.752081] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=20] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=14558208) [2025-05-15 09:40:27.752105] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=24, size_used=0, mem_used=0) [2025-05-15 09:40:27.752140] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=1] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:27.758815] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.759701] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.759718] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.759726] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.768927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.769800] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.769816] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.769823] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.779035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.779897] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.779911] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.779918] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.789144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.789997] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.790014] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.790021] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.798839] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.799260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.799565] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.799592] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027799559) [2025-05-15 09:40:27.799615] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302027599522, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.799669] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=12330047548, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.799687] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.799708] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.799722] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.799732] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.799750] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12330047548, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.799763] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.799777] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.799790] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.799803] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.799813] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.799824] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.799831] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027799643) [2025-05-15 09:40:27.800092] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.800105] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.800115] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.806262] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [5604][T1_TimerWK4_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.809367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.810191] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.810206] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.810213] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.810226] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.811668] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.811690] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.811707] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.811722] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.811734] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.811744] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.811756] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.811766] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.811776] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302027811647, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.811787] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.811796] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.811854] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.811865] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.811876] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.811885] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.819473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.819477] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=18] gc stale ls task succ [2025-05-15 09:40:27.820308] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.820323] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.820330] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.821357] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.821389] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027821353) [2025-05-15 09:40:27.821415] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302027621309, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.821440] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12330093122, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.821456] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.821476] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.821502] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.821512] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.821526] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.821535] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.821544] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027821427) [2025-05-15 09:40:27.821652] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12329920629, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.821673] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.821693] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:27.821704] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.821713] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.821726] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.821734] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.821742] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027821639) [2025-05-15 09:40:27.821843] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4463][T1001_TimerWK3_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=18] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:27.823364] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:27.829325] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1747302027829322, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-05-15 09:40:27.829590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.830408] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.830424] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.830431] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.835208] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=35] PNIO [ratelimit] time: 1747302027835205, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-05-15 09:40:27.839701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.840510] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.840527] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.840534] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.846496] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.846517] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16425207, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.846532] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16425207, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.846546] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.846566] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.846579] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.846590] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.846606] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.846621] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027846620, replica_locations:[]}) [2025-05-15 09:40:27.846643] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.846669] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.847766] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.847787] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.847802] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.847824] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.847838] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.847849] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.847864] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.847877] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027847876, replica_locations:[]}) [2025-05-15 09:40:27.847934] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16423747, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=183, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.849812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.850610] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.850624] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.850631] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.852979] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:27.859923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.860708] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.860724] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.860731] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.861267] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] Garbage Collector is running(seq_=151, gc_interval=10000000) [2025-05-15 09:40:27.861303] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.861333] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=0] gc_check_member_list_ cost time(ret=0, time_us=46) [2025-05-15 09:40:27.861349] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=1) [2025-05-15 09:40:27.861361] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-15 09:40:27.861377] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=13] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-15 09:40:27.861391] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=14) [2025-05-15 09:40:27.861399] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] ObSafeDestroyHandler start process [2025-05-15 09:40:27.867049] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4462][T1001_TimerWK2_Flush][T1001][Y0-0000000000000000-0-0] [lt=12] ====== traversal_flush timer task ====== [2025-05-15 09:40:27.867230] INFO [MDS] flush (mds_table_mgr.cpp:184) [4462][T1001_TimerWK2_Flush][T1001][YB427F000001-000635291DBC6450-0-0] [lt=17] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, {tablet_id:{id:60157}, rec_scn:{val:1747274416328312026, v:0}}, {tablet_id:{id:403}, rec_scn:{val:1747274416328312027, v:0}}, {tablet_id:{id:101079}, rec_scn:{val:1747274416328312028, v:0}}, {tablet_id:{id:50307}, rec_scn:{val:1747274416328312029, v:0}}, {tablet_id:{id:50272}, rec_scn:{val:1747274416328312030, v:0}}, {tablet_id:{id:383}, rec_scn:{val:1747274416328312031, v:0}}, {tablet_id:{id:238}, rec_scn:{val:1747274416328312032, v:0}}, {tablet_id:{id:60306}, rec_scn:{val:1747274416328312033, v:0}}, {tablet_id:{id:50333}, rec_scn:{val:1747274416328312034, v:0}}, {tablet_id:{id:150}, rec_scn:{val:1747274416328312035, v:0}}, {tablet_id:{id:50238}, rec_scn:{val:1747274416328312036, v:0}}, {tablet_id:{id:406}, rec_scn:{val:1747274416328312037, v:0}}, {tablet_id:{id:50531}, rec_scn:{val:1747274416328312038, v:0}}, {tablet_id:{id:1}, rec_scn:{val:1747274416328312039, v:0}}, {tablet_id:{id:60330}, rec_scn:{val:1747274416328312040, v:0}}, {tablet_id:{id:50319}, rec_scn:{val:1747274416328312041, v:0}}, {tablet_id:{id:50377}, rec_scn:{val:1747274416328312042, v:0}}, {tablet_id:{id:396}, rec_scn:{val:1747274416328312043, v:0}}, {tablet_id:{id:467}, rec_scn:{val:1747274416328312044, v:0}}, {tablet_id:{id:60407}, rec_scn:{val:1747274416328312045, v:0}}, {tablet_id:{id:50430}, rec_scn:{val:1747274416328312046, v:0}}, {tablet_id:{id:50104}, rec_scn:{val:1747274416328312047, v:0}}, {tablet_id:{id:50425}, rec_scn:{val:1747274416328312048, v:0}}, {tablet_id:{id:50480}, rec_scn:{val:1747274416328312049, v:0}}, {tablet_id:{id:60503}, rec_scn:{val:1747274416328312050, v:0}}, {tablet_id:{id:60390}, rec_scn:{val:1747274416328312051, v:0}}, {tablet_id:{id:50436}, rec_scn:{val:1747274416328312052, v:0}}, {tablet_id:{id:50160}, rec_scn:{val:1747274416328312053, v:0}}, {tablet_id:{id:474}, rec_scn:{val:1747274416328312054, v:0}}, {tablet_id:{id:60393}, rec_scn:{val:1747274416328312055, v:0}}, {tablet_id:{id:50451}, rec_scn:{val:1747274416328312056, v:0}}, }, max_consequent_callbacked_scn={val:1747289697900328000, v:0}, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:27.867388] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4462][T1001_TimerWK2_Flush][T1001][YB427F000001-000635291DBC6450-0-0] [lt=154] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-15 09:40:27.870034] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.870806] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.870821] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.870828] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.880144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.880905] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.880920] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.880927] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.890253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.891005] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.891021] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.891028] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.895097] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:27.898078] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:27.898078] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=38] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:27.898900] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.899629] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:27.899669] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=12330147991, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.899689] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.899716] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.899733] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.899745] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.899762] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12330147991, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.899775] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.899789] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.899802] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.899812] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.899821] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.899832] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.899839] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027899651) [2025-05-15 09:40:27.900363] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.901105] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.901119] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.901125] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.910475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.911202] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.911217] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.911225] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.911270] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=39] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:27.911550] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.911799] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.911817] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.911840] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.911861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.911873] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.911885] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.911895] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.911908] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302027911770, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.911921] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.911931] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:27.912000] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:27.912010] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:27.912020] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:27.912029] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:27.920592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.921350] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.921365] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.921372] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.921432] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12330193569, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:27.921457] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.921483] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:27.921518] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.921534] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.921553] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.921565] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:27.921578] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027921414) [2025-05-15 09:40:27.921697] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.921723] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027921691) [2025-05-15 09:40:27.921741] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302027721642, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.921763] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12330021076, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:27.921776] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.921792] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:27.921806] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.921816] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.921828] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.921836] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:27.921844] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027921753) [2025-05-15 09:40:27.930501] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [4308][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=7] finish to resize log external storage handler(current_ts=1747302027930499, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-15 09:40:27.930527] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [4308][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=23] snapshot_traffic_info once [2025-05-15 09:40:27.930699] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.931018] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4802646094, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:27.931046] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:27.931058] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1aaefec3a0) [2025-05-15 09:40:27.931070] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4802646094, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:27.931445] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.931460] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.931472] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.937463] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:27.937512] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:27.940810] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.941549] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.941568] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.941578] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.948055] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.948076] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16323641, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.948092] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16323641, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.948116] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.948129] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.948140] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.948155] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.948166] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027948165, replica_locations:[]}) [2025-05-15 09:40:27.948180] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:27.948203] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:27.949390] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:27.949413] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.949427] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:27.949448] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.949461] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:27.949472] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:27.949485] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:27.949505] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302027949504, replica_locations:[]}) [2025-05-15 09:40:27.949564] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=16322117, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=184, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:27.950921] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.951664] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.951682] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.951696] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.956010] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:27.956039] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:27.956056] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=15][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:27.956073] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:27.956088] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:27.956100] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:27.956112] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:27.956124] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:27.956137] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:27.956145] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=7][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:27.956155] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:27.956168] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:27.956180] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:27.956189] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:27.956212] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:27.956223] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:27.956236] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:27.956248] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:27.956261] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:27.956275] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:27.956286] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:27.956306] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:27.956329] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:27.956340] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:27.956346] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:27.956363] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:27.956377] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623E-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:27.956392] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4535][T1002_Occam][T1002][YB427F000001-000635291FAC623E-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:27.956404] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:27.956411] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:27.956421] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:27.956432] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302027955773, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:27.956445] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:27.956456] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, tenant_id=1, columns=0x7f1a84e563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:27.956475] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:27.956561] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1a84e563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:27.956578] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:27.956592] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:27.956600] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:27.956622] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:27.957630] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:27.957657] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-05-15 09:40:27.957669] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:27.957687] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:27.957703] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11790848054, flush_io_finished_ret_:0, flush_io_finished_round_:1502, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:27.959340] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:41, tid:3961}]) [2025-05-15 09:40:27.961031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.961778] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.961796] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.961812] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.971146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.971894] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.971911] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.971926] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.981255] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.982007] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.982024] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.982033] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.991366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:27.992112] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.992131] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:27.992145] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:27.995134] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCE-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.995152] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCE-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.995175] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCE-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.995191] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCE-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.995202] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CCE-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.995213] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:27.995225] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.995235] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.995245] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302027995111, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:27.995258] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.995269] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:27.995341] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:27.995624] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.995642] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:27.995653] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.995663] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:27.995674] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:27.995684] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:27.995698] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:27.995712] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:27.995753] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:27.996579] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCF-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.996598] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCF-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:27.996609] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCF-0-0] [lt=7][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:27.996621] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CCF-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:27.996629] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CCF-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:27.996639] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:27.996648] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:27.996657] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:27.996666] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302027996570, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:27.996677] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:27.996685] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:27.996724] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:27.998142] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=17] gc stale ls task succ [2025-05-15 09:40:27.998955] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:27.999687] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:27.999719] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302027999681) [2025-05-15 09:40:27.999740] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302027799642, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:27.999781] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12330248450, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.999801] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.999827] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.999847] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:27.999862] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:27.999881] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12330248450, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:27.999898] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:27.999915] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:27.999931] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:27.999944] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:27.999960] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.999973] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:27.999986] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302027999760) [2025-05-15 09:40:28.001476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.002218] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.002247] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.002260] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:28.002273] WDIAG [STORAGE.TRANS] get_gts (ob_gts_source.cpp:213) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] get gts leader fail(tmp_ret=-4721, tenant_id=1002) [2025-05-15 09:40:28.002287] WDIAG [MVCC] study (ob_multi_version_garbage_collector.cpp:319) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4012] study min unallocated GTS failed(ret=-4012) [2025-05-15 09:40:28.002301] INFO [MVCC] study (ob_multi_version_garbage_collector.cpp:392) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] study multi version garabage collector end(ret=-4012, this={this:0x7f1a8fcef030, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1747301787989208, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, min_unallocated_GTS={val:4611686018427387903, v:0}, min_unallocated_GTS={val:4611686018427387903, v:0}, max_committed_txn_version={val:4611686018427387903, v:0}, min_active_txn_version={val:4611686018427387903, v:0}) [2025-05-15 09:40:28.002326] WDIAG [LIB] ~ObTimeGuard (utility.h:901) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4389] destruct(*this=time guard 'study' cost too much time, used=1002458, time_dist: study_min_unallocated_GTS=1002422, study_min_unallocated_WRS=0, study_max_commited_txn_version=0, study_min_active_txn_version=1, report=0) [2025-05-15 09:40:28.002345] WDIAG [MVCC] repeat_study (ob_multi_version_garbage_collector.cpp:186) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4012] repeat study failed, we will retry immediately(ret=-4012, this={this:0x7f1a8fcef030, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1747301787989208, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, current_timestamp=1747302026999877) [2025-05-15 09:40:28.003702] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4641][T1002_Occam][T1001][YB427F000001-0006352920DC612A-0-0] [lt=18][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.003721] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4641][T1002_Occam][T1001][YB427F000001-0006352920DC612A-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.003739] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4641][T1002_Occam][T1001][YB427F000001-0006352920DC612A-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:" select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.003758] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4641][T1002_Occam][T1001][YB427F000001-0006352920DC612A-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.003772] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4641][T1002_Occam][T1002][YB427F000001-0006352920DC612A-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:" select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.003788] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 ) [2025-05-15 09:40:28.003801] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.003814] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.003827] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302028003684, sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 ) [2025-05-15 09:40:28.003841] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.003854] WDIAG [MVCC] collect (ob_multi_version_garbage_collector.cpp:678) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql read fail(ret=-5627, ret="OB_SCHEMA_EAGAIN", meta_tenant_id=1001, tenant_id=1002, sql= select snapshot_version, snapshot_type, svr_ip, svr_port, create_time, status from __all_reserved_snapshot where tenant_id = 1002 ) [2025-05-15 09:40:28.003905] WDIAG [MVCC] refresh_ (ob_multi_version_garbage_collector.cpp:534) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] collect snapshot info sql failed(ret=-5627, this={this:0x7f1a8fcef030, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1747301787989208, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, collector={reserved_snapshot_version:{val:4611686018427387903, v:0}, reserved_snapshot_type:0, reserved_status:-1, reserved_create_time:0, reserved_addr:"0.0.0.0:0", is_this_server_disabled:false, status:0}) [2025-05-15 09:40:28.003931] WDIAG [MVCC] repeat_refresh (ob_multi_version_garbage_collector.cpp:225) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5627] repeat refresh failed, we will retry immediately(ret=-5627, this={this:0x7f1a8fcef030, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1747301787989208, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, current_timestamp=1747302028002362) [2025-05-15 09:40:28.003951] INFO [MVCC] repeat_reclaim (ob_multi_version_garbage_collector.cpp:266) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] skip repeat reclaim(ret=0, this={this:0x7f1a8fcef030, last_study_timestamp:0, last_refresh_timestamp:0, last_reclaim_timestamp:1747301787989208, last_sstable_overflow_timestamp:0, has_error_when_study:true, refresh_error_too_long:false, has_error_when_reclaim:false, gc_is_disabled:false, global_reserved_snapshot:{val:0, v:0}, is_inited:true}, current_timestamp=1747302028003950, GARBAGE_COLLECT_EXEC_INTERVAL=600000000) [2025-05-15 09:40:28.003971] WDIAG [OCCAM] operator() (ob_occam_timer.h:85) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4389] run this task use too much time(*p_task_={this:0x7f1abd0aa950, caller:ob_multi_version_garbage_collector.cpp:start:115, function_type:ZN9oceanbase19concurrency_control30ObMultiVersionGarbageCollector5startEvE6$_1152, timer_running_flag:1, total_running_count:1, func_shared_ptr_.ptr:0x7f1abd0aac30, time_wheel:0x7f1ab1029990, thread_pool:0x7f1ab1029170, is_running:0, time_interval:60.00s, expected_run_time:09:41:25.911, task_priority:1, with_handle_protected:1}) [2025-05-15 09:40:28.004018] WDIAG [OCCAM] ~ObOccamFastTimeGuard (ob_occam_time_guard.h:408) [4641][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=0] cost too much time:ob_occam_timer.h:bool oceanbase::common::occam::ObOccamTimerTask::TaskWrapper::operator()(), (this=|threshold=1.00s|start at 09:40:28.402|total=1.00s) [2025-05-15 09:40:28.004217] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18] ====== tenant freeze timer task ====== [2025-05-15 09:40:28.011590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.011740] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.011764] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.011784] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.011799] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.011812] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.011823] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.011832] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.011842] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.011851] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302028011721, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.011864] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.011872] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.011922] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.011934] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.011943] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.011953] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.021474] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.021511] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028021469) [2025-05-15 09:40:28.021530] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302027821426, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.021569] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12330292966, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.021591] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.021617] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:28.021633] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.021644] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.021659] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.021670] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.021681] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028021552) [2025-05-15 09:40:28.021698] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.021761] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12330120473, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.021783] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.021802] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:28.021817] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.021829] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.021844] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.021851] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.021861] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028021748) [2025-05-15 09:40:28.023453] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-05-15 09:40:28.031807] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.035874] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4460][T1001_TimerWK0_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=0] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:28.035899] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4460][T1001_TimerWK0_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=19] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=67) [2025-05-15 09:40:28.041915] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.046083] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:28.046088] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC965F-0-0] [lt=27][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302028046064}) [2025-05-15 09:40:28.046115] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:28.046148] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:28.046166] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:28.046538] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.046553] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:28.046564] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:28.049700] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.049723] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16222001, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.049739] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16222001, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.049753] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.049766] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=16222001, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.049903] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=138][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.049920] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.049936] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.049949] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.049966] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.049981] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.049997] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028049996, replica_locations:[]}) [2025-05-15 09:40:28.050015] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.050037] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.050048] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.050071] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.050114] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11791940594, cache_obj->added_lc()=false, cache_obj->get_object_id()=15707, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.051173] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=52] new mysql sessid created(conn.sessid_=3221552660, support_ssl=false) [2025-05-15 09:40:28.051238] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35] sm conn init succ(conn.sessid_=3221552660, sess.client_addr_="172.22.0.1:42382") [2025-05-15 09:40:28.051260] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552660, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.051385] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.051406] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.051421] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.051433] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.051443] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.051456] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.051470] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.051483] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.051506] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.051517] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.051531] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.051524] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.051545] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028051543, replica_locations:[]}) [2025-05-15 09:40:28.051583] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=29] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.051613] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16220068, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=185, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.051611] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.051635] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.051653] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.051665] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.051692] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.051709] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552660, proxy_sessid=0) [2025-05-15 09:40:28.051734] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=8][errcode=0] server close connection(sessid=3221552660, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.051747] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552660, proxy_sessid=0) [2025-05-15 09:40:28.051755] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.051766] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906B-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552660, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.051823] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552660, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028051461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.051851] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552660, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.051860] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552660, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028051461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.051881] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552660) [2025-05-15 09:40:28.051899] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552660, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.052026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.062139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.065406] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4578][T1002_TimerWK1_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=22] ====== check clog disk timer task ====== [2025-05-15 09:40:28.065431] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4578][T1002_TimerWK1_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:28.067483] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:28.067521] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.067612] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=91][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:28.067633] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:28.067644] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:28.067725] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=80][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:28.072253] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.082368] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.083996] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4460][T1001_TimerWK0_][T1001][Y0-0000000000000000-0-0] [lt=12] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:28.084021] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4460][T1001_TimerWK0_][T1001][Y0-0000000000000000-0-0] [lt=20] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:28.084032] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4460][T1001_TimerWK0_][T1001][Y0-0000000000000000-0-0] [lt=9] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=82) [2025-05-15 09:40:28.086734] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=39] new mysql sessid created(conn.sessid_=3221552661, support_ssl=false) [2025-05-15 09:40:28.086789] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221552661, sess.client_addr_="172.22.0.1:42388") [2025-05-15 09:40:28.086808] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552661, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087037] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.087071] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.087119] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221552662, support_ssl=false) [2025-05-15 09:40:28.087104] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.087135] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=30][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.087151] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.087155] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552662, sess.client_addr_="172.22.0.1:42400") [2025-05-15 09:40:28.087170] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=19][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.087169] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552662, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087162] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552663, support_ssl=false) [2025-05-15 09:40:28.087192] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087208] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552661, proxy_sessid=0) [2025-05-15 09:40:28.087237] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=11][errcode=0] server close connection(sessid=3221552661, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087236] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=38] sm conn init succ(conn.sessid_=3221552663, sess.client_addr_="172.22.0.1:42406") [2025-05-15 09:40:28.087255] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552661, proxy_sessid=0) [2025-05-15 09:40:28.087265] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:28.087262] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552663, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087281] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552664, support_ssl=false) [2025-05-15 09:40:28.087282] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903C-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552661, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.087311] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.087334] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221552664, sess.client_addr_="172.22.0.1:42414") [2025-05-15 09:40:28.087341] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.087354] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552664, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087368] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552661, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028086343, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087374] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.087389] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552661, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.087413] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=38][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.087411] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552661, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028086343, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.087435] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=22][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.087433] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552661) [2025-05-15 09:40:28.087438] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.087445] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.087481] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.087463] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221552661, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.087476] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=19] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087509] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.087520] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=41][errcode=-4018] get session fail(ret=-4018, sessid=3221552662, proxy_sessid=0) [2025-05-15 09:40:28.087520] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.087528] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.087544] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.087550] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.087552] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=16][errcode=0] server close connection(sessid=3221552662, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087565] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552662, proxy_sessid=0) [2025-05-15 09:40:28.087559] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.087573] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:28.087572] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087589] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552663, proxy_sessid=0) [2025-05-15 09:40:28.087585] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902D-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552662, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.087591] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.087616] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=13][errcode=0] server close connection(sessid=3221552663, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087627] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552663, proxy_sessid=0) [2025-05-15 09:40:28.087618] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.087641] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:28.087642] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=23][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.087648] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906C-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552663, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.087661] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=18][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.087662] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=54][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552662, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028086343, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087701] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=25] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087704] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552663, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028087405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087717] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552664, proxy_sessid=0) [2025-05-15 09:40:28.087706] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=40] kill and revert session(conn.sessid_=3221552662, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.087728] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552663, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.087738] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=11][errcode=0] server close connection(sessid=3221552664, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.087728] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552662, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028086343, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.087749] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552664, proxy_sessid=0) [2025-05-15 09:40:28.087745] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552663, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028087405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.087752] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552662) [2025-05-15 09:40:28.087767] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=18][errcode=-4016] session is null [2025-05-15 09:40:28.087767] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552663) [2025-05-15 09:40:28.087776] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903D-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552664, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.087777] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221552662, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.087793] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552663, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.087841] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552664, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302028087405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.087893] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=49] kill and revert session(conn.sessid_=3221552664, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.087915] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552664, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302028087405, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.087946] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] mark session id unused(conn.sessid_=3221552664) [2025-05-15 09:40:28.087974] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552664, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.088463] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] new mysql sessid created(conn.sessid_=3221552665, support_ssl=false) [2025-05-15 09:40:28.088499] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552666, support_ssl=false) [2025-05-15 09:40:28.088528] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552665, sess.client_addr_="172.22.0.1:42428") [2025-05-15 09:40:28.088552] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221552666, sess.client_addr_="172.22.0.1:42432") [2025-05-15 09:40:28.088549] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552665, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.088571] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552666, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.088702] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221552667, support_ssl=false) [2025-05-15 09:40:28.088744] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552667, sess.client_addr_="172.22.0.1:42436") [2025-05-15 09:40:28.088759] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.088795] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.088770] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552667, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.088812] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.088830] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.088838] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.088840] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=51] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.088845] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.088863] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.088870] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.088881] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552665, proxy_sessid=0) [2025-05-15 09:40:28.088888] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.088900] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=8][errcode=0] server close connection(sessid=3221552665, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.088907] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.088911] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552665, proxy_sessid=0) [2025-05-15 09:40:28.088915] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.088919] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.088924] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.088929] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902E-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552665, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.088938] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.088940] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.088953] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552666, proxy_sessid=0) [2025-05-15 09:40:28.088958] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.088962] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.088973] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.088975] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=8][errcode=0] server close connection(sessid=3221552666, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.088987] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.088989] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552666, proxy_sessid=0) [2025-05-15 09:40:28.088994] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.088980] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552665, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028088462, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.088998] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:28.089006] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.089006] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552665, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.089008] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906D-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552666, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.089020] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552667, proxy_sessid=0) [2025-05-15 09:40:28.089024] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552665, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028088462, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.089040] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=12][errcode=0] server close connection(sessid=3221552667, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.089045] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552665) [2025-05-15 09:40:28.089048] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552667, proxy_sessid=0) [2025-05-15 09:40:28.089059] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:28.089060] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552665, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.089066] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552666, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028088462, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.089073] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC902F-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552667, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.089090] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552666, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.089107] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552666, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028088462, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.089126] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552666) [2025-05-15 09:40:28.089141] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552666, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.089177] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552667, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302028088462, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.089197] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552667, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.089252] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552667, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302028088462, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.089286] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] mark session id unused(conn.sessid_=3221552667) [2025-05-15 09:40:28.089299] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552667, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.091548] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=33] new mysql sessid created(conn.sessid_=3221552668, support_ssl=false) [2025-05-15 09:40:28.091598] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221552668, sess.client_addr_="172.22.0.1:42444") [2025-05-15 09:40:28.091625] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552668, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.091821] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.091856] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.091883] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.091912] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.091929] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.091937] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.091966] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=19] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.091986] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552668, proxy_sessid=0) [2025-05-15 09:40:28.092019] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=17][errcode=0] server close connection(sessid=3221552668, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.092034] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552668, proxy_sessid=0) [2025-05-15 09:40:28.092042] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.092056] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903E-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552668, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.092119] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552668, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028091637, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.092158] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221552668, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.092173] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552669, support_ssl=false) [2025-05-15 09:40:28.092178] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552668, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028091637, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.092209] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] mark session id unused(conn.sessid_=3221552668) [2025-05-15 09:40:28.092214] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552669, sess.client_addr_="172.22.0.1:42460") [2025-05-15 09:40:28.092232] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552669, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.092226] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552668, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.092275] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221552670, support_ssl=false) [2025-05-15 09:40:28.092317] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221552670, sess.client_addr_="172.22.0.1:42474") [2025-05-15 09:40:28.092341] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] new mysql sessid created(conn.sessid_=3221552671, support_ssl=false) [2025-05-15 09:40:28.092343] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552670, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.092396] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221552671, sess.client_addr_="172.22.0.1:42476") [2025-05-15 09:40:28.092416] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552671, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.092431] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.092460] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.092474] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.092478] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.092501] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.092510] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.092521] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.092534] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.092547] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552669, proxy_sessid=0) [2025-05-15 09:40:28.092567] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=8][errcode=0] server close connection(sessid=3221552669, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.092562] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.092578] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552669, proxy_sessid=0) [2025-05-15 09:40:28.092585] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.092597] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.092592] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906E-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552669, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.092614] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.092635] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.092644] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.092640] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552669, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028091637, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.092658] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.092668] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552669, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.092672] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.092679] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552669, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028091637, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.092686] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552670, proxy_sessid=0) [2025-05-15 09:40:28.092699] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552669) [2025-05-15 09:40:28.092707] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=11][errcode=0] server close connection(sessid=3221552670, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.092719] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552670, proxy_sessid=0) [2025-05-15 09:40:28.092715] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552669, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.092718] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.092733] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:28.092741] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9030-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552670, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.092777] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.092777] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=30] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.092791] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.092788] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552670, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028091637, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.092800] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.092813] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.092816] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552670, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.092824] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.092825] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552670, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028091637, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.092837] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552671, proxy_sessid=0) [2025-05-15 09:40:28.092842] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552670) [2025-05-15 09:40:28.092862] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=15][errcode=0] server close connection(sessid=3221552671, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.092860] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552670, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.092875] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552671, proxy_sessid=0) [2025-05-15 09:40:28.092883] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.092889] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9031-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552671, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.092946] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552671, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302028092696, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.092987] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=38] kill and revert session(conn.sessid_=3221552671, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.093020] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552671, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302028092696, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.093057] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] mark session id unused(conn.sessid_=3221552671) [2025-05-15 09:40:28.093090] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] connection close(sessid=3221552671, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.095397] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:28.099003] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.099768] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12330347851, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.099799] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.099830] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.099851] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.099866] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.099888] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12330347851, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.099904] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.099922] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.099938] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:28.099952] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:28.099969] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.099982] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.099994] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028099744) [2025-05-15 09:40:28.102586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.106226] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=32] ====== checkpoint timer task ====== [2025-05-15 09:40:28.106980] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=119] succ to get rec scn(*this={this:0x7f1a94c04030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300521088740, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1007}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:28.107024] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=44] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a95378080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a95378080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:28.107204] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=34] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1747274416328312025, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, }, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:28.107299] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=91] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1a979cc1b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1a979d7010, memtable_head:0, memtable_tail:1, t3m:0x7f1aaa0ea030, tables:[0x7f1a952dc080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1a979d8d10, ls_tablet_svr:0x7f1a979cc190}, min_rec_scn={val:1747274408663046000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:28.107335] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=29] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-15 09:40:28.107352] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:28.107364] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:28.107372] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:28.107389] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:28.107417] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=26] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:28.107434] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274408663046000, v:0}, checkpoint_scn_in_ls_meta={val:1747274408663046000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:28.107458] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:5167067136}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:5167067136}, lsn={lsn:5167067136}) [2025-05-15 09:40:28.107495] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:5167067136}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:28.107508] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4460][T1001_TimerWK0_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=11] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-15 09:40:28.111931] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.111954] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.111975] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.111992] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.112006] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.112019] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.112032] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.112043] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.112055] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302028111895, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.112069] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.112081] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.112161] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.112173] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.112185] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.112195] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.112694] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.114229] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:28.114248] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:28.114257] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=7] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:28.114273] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=15] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:28.119685] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=17] gc stale ls task succ [2025-05-15 09:40:28.121545] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12330393403, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.121570] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.121655] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=82][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:28.121675] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.121688] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.121704] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.121716] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.121727] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028121529) [2025-05-15 09:40:28.121812] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.121838] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028121805) [2025-05-15 09:40:28.121854] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302027921752, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.121879] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12330220910, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.121895] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.121913] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:28.122803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.126864] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:28.126893] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:28.126907] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:28.126926] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:28.126942] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11791017285, flush_io_finished_ret_:0, flush_io_finished_round_:1502, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:28.132910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.143016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.151747] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.151768] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16119952, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.151857] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=89][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16119952, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.151871] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.151882] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=16119952, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.151897] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.151912] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.151927] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.151942] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.151953] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.151965] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.151981] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028151979, replica_locations:[]}) [2025-05-15 09:40:28.151997] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.152022] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.152033] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.152072] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.152120] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792042596, cache_obj->added_lc()=false, cache_obj->get_object_id()=15708, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.153124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.153261] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.153283] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.153299] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.153312] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.153321] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.153333] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.153346] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.153359] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.153373] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.153383] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.153398] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.153412] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028153411, replica_locations:[]}) [2025-05-15 09:40:28.153478] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=16118204, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=186, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.159761] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:746, tid:4641}]) [2025-05-15 09:40:28.163234] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.169978] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:28.170009] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:28.170021] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:28.170036] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:28.170047] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:28.170059] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:28.170076] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:28.170087] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:28.170098] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:28.170108] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=9][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:28.170120] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:28.170131] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:28.170142] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:28.170152] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:28.170174] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:28.170186] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.170199] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.170211] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:28.170223] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:28.170240] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:28.170247] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:28.170267] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:28.170288] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:28.170298] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:28.170304] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:28.170322] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:28.170336] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C643D-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.170348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:28.170360] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.170372] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:28.170382] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:28.170394] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302028169727, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.170408] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=14][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:28.170420] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.170505] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=8][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:28.170520] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=13][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:28.170529] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=8][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:28.170540] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:28.170553] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:28.170567] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:28.170578] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C643D-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:28.173348] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.174011] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.174035] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=88] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.174190] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.174901] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.183456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.190690] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=24] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:28.193908] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.199064] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.199810] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001) [2025-05-15 09:40:28.199831] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.199858] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14] ====== tenant freeze timer task ====== [2025-05-15 09:40:28.199953] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=121][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028199804) [2025-05-15 09:40:28.199969] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302027999759, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.200018] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] get wrs ts(ls_id={id:1}, delta=12330448237, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.200037] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.200067] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12330448237, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.200080] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.204580] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.209764] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=16][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:28.209784] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.209796] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=8][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.209814] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.209829] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.209838] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.209853] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.209870] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.209884] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.209899] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.209911] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.209927] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.209954] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.209970] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=12] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:28.211563] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:28.211589] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:28.211600] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:28.211611] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:28.211628] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=15][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:28.211635] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:28.211651] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:28.211661] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:28.211669] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=6][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:28.211680] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:28.211688] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:28.211699] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:28.211707] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:28.211716] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:28.211733] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:28.211743] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.211752] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=8][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.211763] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:28.211771] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:28.211781] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=9][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:28.211793] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:28.211812] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:28.211833] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:28.211845] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=12][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:28.211851] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:28.211868] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:28.211882] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.211899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=17][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:28.211915] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.211929] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:28.211938] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:28.211948] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302028211353, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.211961] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:28.211969] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.211956] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.211980] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.212004] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.212021] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.212037] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.212051] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=8][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:28.212052] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.212064] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=12][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:28.212067] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.212073] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=8][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:28.212082] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=8][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:28.212080] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.212093] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:28.212096] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302028211930, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.212102] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=9][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:28.212110] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638C-0-0] [lt=7][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:28.212110] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.212122] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.212189] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.212201] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.212216] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.212230] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.214692] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.217004] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.217034] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.217053] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.218038] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:28.218059] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.218074] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.218086] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:28.218097] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:28.218109] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:28.221595] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.221626] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028221588) [2025-05-15 09:40:28.221659] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302028021544, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.221691] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12330493805, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.221710] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.221733] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.221877] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12330321312, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.221898] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.221916] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.223547] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-05-15 09:40:28.224803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.228901] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:28.228923] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.228989] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=65][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.229006] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302028227989, register_time_us=-1, curr_time_us=1747302028229004) [2025-05-15 09:40:28.229024] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:28.229035] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.234913] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.245022] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.252196] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.252218] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:28.252227] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:28.252242] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:28.252252] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:28.252289] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:28.253608] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.253631] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16018088, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.253646] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=16018088, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.253660] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.253669] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=16018088, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.253683] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.253698] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.253712] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.253726] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.253738] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.253753] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.253770] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028253768, replica_locations:[]}) [2025-05-15 09:40:28.253788] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.253807] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:186, local_retry_times:186, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:28.253824] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.253836] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.253850] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:28.253859] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:28.253866] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=5][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:28.253881] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:28.253893] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.253928] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792144406, cache_obj->added_lc()=false, cache_obj->get_object_id()=15709, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.255052] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.255081] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.255117] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.255130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.255131] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.255148] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.255162] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.255170] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.255183] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.255198] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.255212] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.255226] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.255237] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.255251] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.255265] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028255265, replica_locations:[]}) [2025-05-15 09:40:28.255282] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.255296] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.255308] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.255324] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:28.255335] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:28.255346] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:28.255364] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:28.255379] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:28.255390] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:28.255402] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:28.255412] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:28.255424] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:28.255438] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:28.255450] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:28.255458] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:28.255469] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:28.255479] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:28.255501] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:28.255510] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:28.255527] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:28.255539] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.255550] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:28.255562] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:28.255574] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:28.255584] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=187, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:28.255609] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=16016073, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=187, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.257278] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=37] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:28.257306] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-05-15 09:40:28.257318] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:28.257341] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:28.257372] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=31] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11791147698, flush_io_finished_ret_:0, flush_io_finished_round_:1502, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:28.262997] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=28] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:28.265244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.275358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.285465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.295574] INFO [SERVER] refresh_cpu_frequency (ob_server.cpp:3776) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=41] Cpu frequency changed(from=2452857, to=2453669) [2025-05-15 09:40:28.295582] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.295627] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20] === Run print tenant memory usage task === [2025-05-15 09:40:28.295721] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=55] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 106078208 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9223372036854775807 mem_tenant_hold= 580640768 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1073741824 mem_tenant_hold= 36556800 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1073741824 mem_tenant_hold= 23973888 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1747302028295526 active_memstore_used= 119537664 total_memstore_used= 119537664 total_memstore_hold= 119537664 memstore_freeze_trigger_limit= 429496720 memstore_limit= 2147483640 mem_tenant_limit= 5368709120 mem_tenant_hold= 850857984 max_mem_memstore_can_get_now= 2630414336 memstore_alloc_pos= 119537664 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1001 now= 1747302028295526 active_memstore_used= 73400320 total_memstore_used= 73400320 total_memstore_hold= 73400320 memstore_freeze_trigger_limit= 345544580 memstore_limit= 1727722920 mem_tenant_limit= 4319307366 mem_tenant_hold= 699817984 max_mem_memstore_can_get_now= 1965982470 memstore_alloc_pos= 73400320 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1002 now= 1747302028295526 active_memstore_used= 188743680 total_memstore_used= 188743680 total_memstore_hold= 188743680 memstore_freeze_trigger_limit= 3887376620 memstore_limit= 19436883100 mem_tenant_limit= 38873766298 mem_tenant_hold= 971210752 max_mem_memstore_can_get_now= 36023507354 memstore_alloc_pos= 188743680 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-05-15 09:40:28.295849] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=17] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:28.295862] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=25] [CHUNK_MGR] limit= 52856746803 hold= 3215794176 total_hold= 3428843520 used= 3160961024 freelists_hold= 54833152 total_maps= 1125 total_unmaps= 43 large_maps= 131 large_unmaps= 0 huge_maps= 48 huge_unmaps= 38 resident_size= 0 unmanaged_memory_size= 106078208 virtual_memory_used= 4345532416 [OS_PARAMS] vm.max_map_count= 65530 transparent_hugepages= always [CHUNK_MGR] 2 MB_CACHE: hold= 23068672 count= 11 pushes= 2800955 pops= 2800944 maps= 946 unmaps= 5 [CHUNK_MGR] 4 MB_CACHE: hold= 31764480 count= 15 pushes= 1195 pops= 1180 maps= 69 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 21 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 18 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 7 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [2025-05-15 09:40:28.295897] INFO print (ob_malloc_time_monitor.cpp:37) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-15 09:40:28.295990] WDIAG load_file_to_string (utility.h:673) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:28.296006] WDIAG get_ethernet_speed (utility.cpp:575) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:28.296017] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:28.296029] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:28.299121] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.299887] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12330548559, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.299911] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.299947] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.299986] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12330548559, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.300016] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.301185] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5302, clean_start_pos=11800000, clean_num=200000) [2025-05-15 09:40:28.305688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.312026] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.312046] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.312068] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.312085] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.312097] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.312109] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.312121] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.312132] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.312144] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302028311992, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.312157] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.312168] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.312241] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.312252] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.312262] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.312273] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.312912] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.312934] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.312945] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:28.312970] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:28.315797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.317060] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.317084] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.317096] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.318094] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:28.318114] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.318125] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.318140] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:28.318151] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:28.318162] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:28.321674] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12330593055, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.321706] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.321927] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.321966] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028321921) [2025-05-15 09:40:28.321982] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302028121869, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.322016] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12330421617, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.322032] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.325907] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.328961] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:28.328983] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.328999] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.336017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.346124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.351794] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-05-15 09:40:28.351827] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.351837] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:28.351846] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:28.351856] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:28.351936] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:28.351954] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.351960] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:28.351966] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:28.351975] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:28.352023] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:28.353470] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:28.353551] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=30] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:28.353620] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:28.353682] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:28.353711] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:15027, total:15038) [2025-05-15 09:40:28.353747] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:28.353798] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:15027, total:15038) [2025-05-15 09:40:28.353826] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:28.353844] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:28.353856] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:28.353877] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628763][2nd=1890]}) [2025-05-15 09:40:28.355731] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.355754] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15915966, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.355770] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15915966, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.355785] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.355797] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=15915966, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.355811] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.355826] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.355838] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.355853] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.355865] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.355880] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.355896] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028355895, replica_locations:[]}) [2025-05-15 09:40:28.355916] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.355934] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:187, local_retry_times:187, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:28.355953] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.355965] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.355982] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:28.355993] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:28.355999] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:28.356032] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:28.356047] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.356092] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792246567, cache_obj->added_lc()=false, cache_obj->get_object_id()=15710, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.356233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.357185] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.357213] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.357245] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.357260] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.357275] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.357289] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.357297] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.357311] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.357326] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.357340] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.357354] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.357362] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.357377] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.357391] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028357390, replica_locations:[]}) [2025-05-15 09:40:28.357408] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.357421] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.357434] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.357449] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:28.357459] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:28.357470] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:28.357496] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:28.357511] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:28.357518] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:28.357529] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:28.357540] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:28.357550] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:28.357559] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:28.357571] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:28.357581] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:28.357592] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:28.357602] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:28.357610] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:28.357621] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:28.357637] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:28.357648] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.357659] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:28.357671] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:28.357684] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:28.357694] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=188, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:28.357717] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] will sleep(sleep_us=100000, remain_us=15913965, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=188, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.359017] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-05-15 09:40:28.360452] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=14][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.360471] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=19][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.360480] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.360501] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=20][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.360514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=12][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:28.360529] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=10][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:28.360542] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.360552] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C665B-0-0] [lt=10][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5157, ret=-5157) [2025-05-15 09:40:28.360565] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:28.360578] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.360590] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:28.360601] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78cb0f0, start=1747302028360434, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:28.360611] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:28.360622] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5157) [2025-05-15 09:40:28.360698] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] failed to fetch max rule version(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.360710] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] failed to fetch max rule version(ret=-5157) [2025-05-15 09:40:28.360717] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] failed to sync rule from inner table(ret=-5157) [2025-05-15 09:40:28.363355] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4291][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=16] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}) [2025-05-15 09:40:28.366342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.373459] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-15 09:40:28.373480] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4289][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=32] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-15 09:40:28.373511] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4289][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=25] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:28.373525] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4289][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:28.373540] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4289][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:28.374923] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2025-05-15 09:40:28.374950] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=24][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2025-05-15 09:40:28.374961] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2025-05-15 09:40:28.374975] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2025-05-15 09:40:28.374989] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:28.375001] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:28.375013] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2025-05-15 09:40:28.375024] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:28.375035] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:28.375044] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:28.375055] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:28.375063] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=7][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:28.375074] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:28.375085] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:28.375106] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:28.375117] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.375129] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.375140] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:28.375147] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=6][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:28.375159] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:28.375165] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:28.375182] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:28.375201] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:28.375210] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:28.375219] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:28.375234] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:28.375247] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.375259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_ReqMemEvict][T1][YB427F000001-000635291C5C638D-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2025-05-15 09:40:28.375271] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:28.375281] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:28.375289] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:28.375299] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca3b0, start=1747302028374724, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:28.375307] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:28.375317] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:28.375377] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2025-05-15 09:40:28.375389] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] failed to fetch max rule version(ret=-5019) [2025-05-15 09:40:28.375395] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2025-05-15 09:40:28.375993] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4289][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=1, alloc_obj_list=[{obj_id:15711, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:1, added_to_lc:false, mem_used:25832}]) [2025-05-15 09:40:28.376014] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4289][T1_TimerWK0_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=19] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:28.377433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.380516] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [3960][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=25] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=3626817093632, fast_delete_log_mode=false) [2025-05-15 09:40:28.386109] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [4291][T1_TimerWK2_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=27] get_replay_process success(id={id:1}, submitted_log_size=125824135, unsubmitted_log_size=125824135, replayed_log_size=125824135, unreplayed_log_size=0) [2025-05-15 09:40:28.386148] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [4291][T1_TimerWK2_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=22] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=119, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=119, replayed_log_size(MB)=119, last_submitted_log_size(MB)=119, last_replayed_log_size(MB)=119, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-15 09:40:28.387542] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.395222] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=92] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.395986] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.396043] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.396063] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.396127] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.396713] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.396852] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:28.397648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.399180] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.399932] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.399959] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028399924) [2025-05-15 09:40:28.399976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302028199998, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.400018] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12330647907, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.400037] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.400070] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1001}, delta=12330647907, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.400084] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.407756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.411664] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:28.411689] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.411702] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:28.411724] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:28.412017] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.412038] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.412060] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.412076] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.412092] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.412107] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.412120] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.412131] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.412139] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302028411992, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.412151] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.412160] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.412222] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.412233] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.412243] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.412253] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.417118] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.417141] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.417152] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.417864] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.418161] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:28.418183] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.418199] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.418216] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:28.418227] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:28.418235] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:28.421711] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.421749] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028421705) [2025-05-15 09:40:28.421767] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302028221670, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.421803] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12330693493, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.421822] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.421993] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12330521000, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.422014] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.426517] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] Cache replace map node details(ret=0, replace_node_count=0, replace_time=2881, replace_start_pos=5600000, replace_num=100000) [2025-05-15 09:40:28.426541] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=10) [2025-05-15 09:40:28.427974] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.429017] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:28.429038] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.429054] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.438085] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.441228] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:1001) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4036] sys tenant schema is not ready(ret=-4036, ret="OB_NEED_RETRY") [2025-05-15 09:40:28.446748] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:28.446787] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:28.448195] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.457857] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.457883] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15813846, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.457901] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15813846, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.457916] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.457930] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=15813846, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.457947] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.457965] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.457980] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.457995] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.458008] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.458024] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.458043] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028458041, replica_locations:[]}) [2025-05-15 09:40:28.458063] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.458083] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:188, local_retry_times:188, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:28.458101] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.458113] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.458129] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:28.458139] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:28.458146] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:28.458163] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:28.458175] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.458217] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792348695, cache_obj->added_lc()=false, cache_obj->get_object_id()=15711, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.458302] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.459458] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.459499] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.459541] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.459557] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.459572] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.459587] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.459598] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.459613] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.459629] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.459642] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.459657] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.459665] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.459679] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.459693] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028459692, replica_locations:[]}) [2025-05-15 09:40:28.459710] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.459724] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.459736] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:28.459752] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:28.459765] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:28.459782] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:28.459803] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:28.459817] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:28.459825] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:28.459837] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:28.459847] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:28.459858] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:28.459871] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:28.459884] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:28.459894] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:28.459905] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:28.459918] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:28.459930] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:28.459941] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:28.459958] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:28.459969] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:28.459978] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:28.459989] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:28.460002] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:28.460012] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=189, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:28.460036] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=15811645, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=189, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.468413] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.478522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.488631] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.495757] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.495779] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.495788] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.495797] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.495808] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:28.495820] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:28.495835] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:28.495848] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:28.495879] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:28.495969] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:28.495990] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.496003] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:28.496022] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:28.496035] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:28.496047] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:28.496059] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:28.496076] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:28.498738] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.498941] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4577][T1002_TimerWK0_TenantMetaMemMg][T1002][Y0-0000000000000000-0-0] [lt=13] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:728, free_obj_hold_cnt:0, allocator used:2795520, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=3002880, tablets_mem_limit=7774753240, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:9, free_obj_hold_cnt:0, allocator used:18432, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=728) [2025-05-15 09:40:28.499260] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.500005] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12330748479, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.500025] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.500064] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12330748479, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.500077] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.501435] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:28.508845] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.512081] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.512103] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.512125] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.512141] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.512154] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.512166] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.512177] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.512187] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.512198] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302028512051, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.512208] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.512219] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.512285] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.512296] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.512305] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.512315] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.517178] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.517203] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.517214] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.518218] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:28.518238] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.518250] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.518269] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:28.518280] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:28.518291] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:28.518955] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.521797] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12330793006, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.521836] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.522047] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.522074] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028522041) [2025-05-15 09:40:28.522091] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302028321995, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.522111] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12330621570, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.522136] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.529067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.529076] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:28.529094] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.529129] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.536288] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:28.541301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.541420] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:28.546997] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.547020] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.547054] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:28.547071] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=9105, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:28.547172] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:28.547194] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:28.547213] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:28.547230] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:28.547591] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.547605] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:28.547615] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:28.551415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.552500] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552681, support_ssl=false) [2025-05-15 09:40:28.552572] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=44] sm conn init succ(conn.sessid_=3221552681, sess.client_addr_="172.22.0.1:42482") [2025-05-15 09:40:28.552592] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552681, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.552772] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.552816] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.552844] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.552868] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.552884] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.552893] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.552919] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.552937] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552681, proxy_sessid=0) [2025-05-15 09:40:28.552969] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=13][errcode=0] server close connection(sessid=3221552681, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.552982] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552681, proxy_sessid=0) [2025-05-15 09:40:28.552991] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:28.552999] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC906F-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552681, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.553051] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552681, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028552610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.553078] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552681, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.553088] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552681, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028552610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.553108] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552681) [2025-05-15 09:40:28.553124] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552681, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.557931] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC668F-0-0] [lt=37][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.557958] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC668F-0-0] [lt=26][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.557974] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC668F-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.558022] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC668F-0-0] [lt=46][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.558034] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.558051] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.558062] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.558070] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.558079] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302028557903, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.558096] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.558105] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:28.558181] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=16][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:28.558198] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=16][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:28.558208] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=9][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:28.558223] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=14][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:28.558234] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=8][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:28.558250] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=14][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:28.558258] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC668F-0-0] [lt=8][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:28.560167] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.560192] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15711534, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.560211] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15711534, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.560231] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.560241] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=15711534, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.560260] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.560281] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.560302] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.560321] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.560331] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.560351] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.560372] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028560370, replica_locations:[]}) [2025-05-15 09:40:28.560400] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.560430] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.560446] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.560470] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.560528] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792451004, cache_obj->added_lc()=false, cache_obj->get_object_id()=15712, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.560585] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:90, tid:4641}]) [2025-05-15 09:40:28.561530] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.561759] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.561782] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.561808] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.561827] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.561835] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.561853] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.561872] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.561886] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.561904] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.561913] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.561932] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.561942] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028561942, replica_locations:[]}) [2025-05-15 09:40:28.562012] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15709670, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=190, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.569691] INFO pn_ratelimit (group.c:653) [3955][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=21] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-15 09:40:28.571647] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.581759] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.588376] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552683, support_ssl=false) [2025-05-15 09:40:28.588471] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=43] sm conn init succ(conn.sessid_=3221552683, sess.client_addr_="172.22.0.1:42492") [2025-05-15 09:40:28.588501] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552683, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.588714] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.588754] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.588793] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.588820] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552684, support_ssl=false) [2025-05-15 09:40:28.588832] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=36][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.588845] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221552685, support_ssl=false) [2025-05-15 09:40:28.588854] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=21][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.588868] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.588879] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552685, sess.client_addr_="172.22.0.1:42522") [2025-05-15 09:40:28.588882] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=31] sm conn init succ(conn.sessid_=3221552684, sess.client_addr_="172.22.0.1:42508") [2025-05-15 09:40:28.588892] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552685, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.588897] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.588902] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552684, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.588921] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552683, proxy_sessid=0) [2025-05-15 09:40:28.588979] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=22][errcode=0] server close connection(sessid=3221552683, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.588999] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552683, proxy_sessid=0) [2025-05-15 09:40:28.589012] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:28.589029] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.589033] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC903F-0-0] [lt=18] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552683, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.589037] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552686, support_ssl=false) [2025-05-15 09:40:28.589054] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.589070] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.589079] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.589089] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=10][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.589082] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552683, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.589090] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221552686, sess.client_addr_="172.22.0.1:42526") [2025-05-15 09:40:28.589103] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221552683, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.589112] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552683, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.589109] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552686, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.589126] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221552683) [2025-05-15 09:40:28.589148] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=59] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.589145] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552683, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.589170] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.589096] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.589196] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.589205] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.589199] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=100][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.589212] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.589229] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.589231] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.589249] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552684, proxy_sessid=0) [2025-05-15 09:40:28.589257] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552685, proxy_sessid=0) [2025-05-15 09:40:28.589276] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=9][errcode=0] server close connection(sessid=3221552684, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.589290] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552684, proxy_sessid=0) [2025-05-15 09:40:28.589297] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:28.589305] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=15][errcode=0] server close connection(sessid=3221552685, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.589319] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552685, proxy_sessid=0) [2025-05-15 09:40:28.589314] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9070-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552684, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.589336] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=17][errcode=-4016] session is null [2025-05-15 09:40:28.589328] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.589349] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9040-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552685, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.589363] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552684, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.589373] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.589385] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552684, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.589396] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552684, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.589417] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552684) [2025-05-15 09:40:28.589418] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552685, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.589440] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552685, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.589416] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.589449] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552685, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.589439] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552684, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.589463] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552685) [2025-05-15 09:40:28.589455] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=38][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.589473] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.589476] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552685, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.589483] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.589524] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=20] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.589548] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552686, proxy_sessid=0) [2025-05-15 09:40:28.589594] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=14][errcode=0] server close connection(sessid=3221552686, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.589608] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552686, proxy_sessid=0) [2025-05-15 09:40:28.589617] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.589629] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9032-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552686, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.589705] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552686, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.589751] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=43] kill and revert session(conn.sessid_=3221552686, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.589773] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552686, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028588610, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.589803] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] mark session id unused(conn.sessid_=3221552686) [2025-05-15 09:40:28.589838] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] connection close(sessid=3221552686, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.590197] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552687, support_ssl=false) [2025-05-15 09:40:28.590236] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552687, sess.client_addr_="172.22.0.1:42530") [2025-05-15 09:40:28.590253] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552687, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.590281] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552688, support_ssl=false) [2025-05-15 09:40:28.590317] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552688, sess.client_addr_="172.22.0.1:42540") [2025-05-15 09:40:28.590333] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552688, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.590438] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=33] new mysql sessid created(conn.sessid_=3221552689, support_ssl=false) [2025-05-15 09:40:28.590463] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.590506] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.590511] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221552689, sess.client_addr_="172.22.0.1:42556") [2025-05-15 09:40:28.590528] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.590531] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.590532] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552689, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.590550] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.590550] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.590561] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.590579] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=17][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.590586] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=31][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.590592] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.590608] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552687, proxy_sessid=0) [2025-05-15 09:40:28.590609] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.590619] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.590625] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=7][errcode=0] server close connection(sessid=3221552687, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.590632] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.590636] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552687, proxy_sessid=0) [2025-05-15 09:40:28.590648] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:28.590646] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.590655] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9071-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552687, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.590662] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552688, proxy_sessid=0) [2025-05-15 09:40:28.590688] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=14][errcode=0] server close connection(sessid=3221552688, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.590704] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552688, proxy_sessid=0) [2025-05-15 09:40:28.590707] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552687, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028589670, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.590725] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=20][errcode=-4016] session is null [2025-05-15 09:40:28.590730] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552687, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.590741] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552687, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028589670, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.590735] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9041-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552688, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.590760] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552687) [2025-05-15 09:40:28.590756] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.590775] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552687, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.590798] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.590811] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552688, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028589670, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.590833] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552688, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.590822] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.590844] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.590857] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.590857] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552688, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028589670, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.590866] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.590878] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552688) [2025-05-15 09:40:28.590884] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.590890] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552688, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.590898] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552689, proxy_sessid=0) [2025-05-15 09:40:28.590922] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=11][errcode=0] server close connection(sessid=3221552689, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.590934] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552689, proxy_sessid=0) [2025-05-15 09:40:28.590944] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:28.590956] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9033-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552689, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.591015] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552689, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028590729, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.591049] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221552689, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.591065] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552689, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028590729, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.591095] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] mark session id unused(conn.sessid_=3221552689) [2025-05-15 09:40:28.591116] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552689, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.591879] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.593087] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552690, support_ssl=false) [2025-05-15 09:40:28.593152] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] sm conn init succ(conn.sessid_=3221552690, sess.client_addr_="172.22.0.1:42564") [2025-05-15 09:40:28.593175] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552690, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.593339] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.593376] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.593394] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.593412] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.593420] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.593433] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.593446] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.593462] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552690, proxy_sessid=0) [2025-05-15 09:40:28.593480] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=8][errcode=0] server close connection(sessid=3221552690, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.593504] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=23][errcode=-4018] get session fail(ret=-4018, sessid=3221552690, proxy_sessid=0) [2025-05-15 09:40:28.593511] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.593527] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9072-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552690, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.593583] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552690, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028592846, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.593616] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221552690, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.593628] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552690, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302028592846, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.593645] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552690) [2025-05-15 09:40:28.593662] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552690, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.593887] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552691, support_ssl=false) [2025-05-15 09:40:28.593950] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221552692, support_ssl=false) [2025-05-15 09:40:28.593962] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36] sm conn init succ(conn.sessid_=3221552691, sess.client_addr_="172.22.0.1:42570") [2025-05-15 09:40:28.593977] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552691, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.594001] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552692, sess.client_addr_="172.22.0.1:42586") [2025-05-15 09:40:28.594014] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552692, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.594125] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552693, support_ssl=false) [2025-05-15 09:40:28.594185] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31] sm conn init succ(conn.sessid_=3221552693, sess.client_addr_="172.22.0.1:42592") [2025-05-15 09:40:28.594186] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.594206] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.594205] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552693, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.594225] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.594238] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.594258] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.594262] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.594289] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=26][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.594290] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=30][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.594304] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.594307] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.594323] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=16][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.594326] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=21][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.594347] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.594349] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.594371] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552691, proxy_sessid=0) [2025-05-15 09:40:28.594369] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552692, proxy_sessid=0) [2025-05-15 09:40:28.594368] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.594404] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.594414] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.594429] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.594437] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.594435] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=12][errcode=0] server close connection(sessid=3221552692, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.594447] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.594449] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552692, proxy_sessid=0) [2025-05-15 09:40:28.594458] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:28.594452] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=9][errcode=0] server close connection(sessid=3221552691, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.594459] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=6] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.594468] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552691, proxy_sessid=0) [2025-05-15 09:40:28.594472] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552693, proxy_sessid=0) [2025-05-15 09:40:28.594466] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9034-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552692, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.594498] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=29][errcode=-4016] session is null [2025-05-15 09:40:28.594504] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=13][errcode=0] server close connection(sessid=3221552693, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.594515] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552693, proxy_sessid=0) [2025-05-15 09:40:28.594522] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.594512] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9042-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552691, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.594529] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552692, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028593906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.594536] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9073-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552693, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.594551] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552692, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.594563] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552692, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302028593906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.594560] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552691, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302028593906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.594582] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552692) [2025-05-15 09:40:28.594584] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552691, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.594593] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552691, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302028593906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.594607] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552691) [2025-05-15 09:40:28.594601] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552693, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028593906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.594601] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552692, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.594629] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552693, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.594627] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552691, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.594643] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552693, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302028593906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.594659] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552693) [2025-05-15 09:40:28.594676] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552693, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.599298] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.600051] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.600091] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028600044) [2025-05-15 09:40:28.600115] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302028399991, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.600146] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12330848004, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.600167] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.600191] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.600209] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.600220] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.600238] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12330848004, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.600253] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.600268] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.600277] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:28.600287] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:28.600301] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.600312] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.600323] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028600126) [2025-05-15 09:40:28.600595] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.600617] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.600629] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to get percent [2025-05-15 09:40:28.600665] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-5627] failed to calculate global bound size(ret=-5627, ids[i]=1001) [2025-05-15 09:40:28.600697] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.600713] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.600721] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.600730] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.600738] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:28.600744] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] failed to get percent [2025-05-15 09:40:28.600749] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-5157] failed to calculate global bound size(ret=-5157, ids[i]=1002) [2025-05-15 09:40:28.601991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.612103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.612201] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.612235] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.614834] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=64][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.614866] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.614877] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.614891] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.614900] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:28.614913] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=12] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-05-15 09:40:28.614929] WDIAG [SQL.PC] run_plan_cache_task (ob_plan_cache.cpp:2766) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] fail to update plan cache memory sys val(ret=-4029) [2025-05-15 09:40:28.614940] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=7] start lib cache evict(tenant_id=1002, mem_hold=2097152, mem_limit=1943688310, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:28.614957] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=15] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=2097152, mem_limit=1943688310, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:28.614973] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=15] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:28.617242] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.617270] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.617281] WDIAG [CLOG] nonblock_get_leader (ob_location_adapter.cpp:127) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] location adapter nonblock_get_leader failed(ret=-4721, id=1, leader="0.0.0.0:0") [2025-05-15 09:40:28.617294] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.617308] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302028617189, register_time_us=-1, curr_time_us=1747302028617306) [2025-05-15 09:40:28.617330] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.617339] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.617353] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:28.617482] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=0, alloc_obj_list=[]) [2025-05-15 09:40:28.617508] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4580][T1002_TimerWK3_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=26] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:28.618283] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.621839] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.621870] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028621830) [2025-05-15 09:40:28.621887] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302028421783, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.621920] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12330893587, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.621939] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.621964] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:28.621984] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.621995] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.622011] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.622020] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.622030] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028621898) [2025-05-15 09:40:28.622101] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:28.622120] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302028622099, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302028622099, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:28.622164] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31] get wrs ts(ls_id={id:1}, delta=12330721094, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.622179] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.622195] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:28.622209] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.622216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.622220] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.622234] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.622240] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.622249] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028622151) [2025-05-15 09:40:28.626627] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=9) [2025-05-15 09:40:28.629148] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.632330] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.634294] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EAB78-0-0] [lt=26][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302028209950}) [2025-05-15 09:40:28.642443] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.654046] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.662159] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.662185] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15609545, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.662202] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15609545, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.662217] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.662231] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=15609545, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.662246] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.662263] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.662281] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.662296] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.662308] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.662324] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.662344] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028662341, replica_locations:[]}) [2025-05-15 09:40:28.662363] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.662390] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.662402] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.662446] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.662509] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792552986, cache_obj->added_lc()=false, cache_obj->get_object_id()=15713, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.663783] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.663806] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.663822] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.663836] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.663846] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.663859] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.663875] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.663890] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.663904] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.663915] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.663927] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.663942] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028663940, replica_locations:[]}) [2025-05-15 09:40:28.664009] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15607673, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=191, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.664161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.666574] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4491][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=34] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1aa57f82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:28.674275] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.678945] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=91] tx gc loop thread is running(MTL_ID()=1001) [2025-05-15 09:40:28.678977] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=31] try gc retain ctx [2025-05-15 09:40:28.678984] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-05-15 09:40:28.679047] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] finish update min start scn(max_decided_scn={val:1747289697900328000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289695031638001, v:0}, keep_alive_scn_:{val:1747289695031638001, v:0}, update_ts_:1747302028678556}) [2025-05-15 09:40:28.679650] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=23] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:28.684387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.694495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.699351] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.700123] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12330948532, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.700148] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.700173] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.700190] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.700203] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.700221] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12330948532, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.700235] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.700256] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.700265] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:28.700275] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:28.700286] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.700297] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.700307] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028700104) [2025-05-15 09:40:28.701726] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=17] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:28.704604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.714708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.716919] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [4309][T1_RFLWorker][T1][YB427F000001-000635291B0C6112-0-0] [lt=26] ObRemoteFetchWorker is running(thread_index=0) [2025-05-15 09:40:28.717309] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.717338] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.717350] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.717362] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", [SlidingWindow]={palf_id:1, self:"127.0.0.1:2882", lsn_allocator:{max_log_id:6576040, max_lsn:2944224980, max_scn:{val:1747289697927760000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:2944224391}, reuse_lsn:{lsn:2944224980}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:2944224980}}, last_submit_lsn:{lsn:2944224391}, last_submit_end_lsn:{lsn:2944224980}, last_submit_log_id:6576040, last_submit_log_pid:4, max_flushed_lsn:{lsn:2944224391}, max_flushed_end_lsn:{lsn:2944224980}, max_flushed_log_pid:4, committed_end_lsn:{lsn:2944224391}, last_slide_log_id:6576039, last_slide_scn:{val:1747289697727835001, v:0}, last_slide_lsn:{lsn:2944224269}, last_slide_end_lsn:{lsn:2944224391}, last_slide_log_pid:4, last_slide_log_accum_checksum:2881944827, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f1ab0f48530}, [StateMgr]={this:0x7f1ab0f4c9b0, self:"127.0.0.1:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:-1, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7f1ab0f53470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"127.0.0.1:2882", alive_paxos_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:2442153277}, prev_mode_pid:4}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:4, config_seq:5}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f1ab0f49c70}, [ModeMgr]={palf_id:1, self:"127.0.0.1:2882", applied_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:2818400256}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, log_config_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:2442153277}, prev_mode_pid:4}, log_snapshot_meta:{version:2, base_lsn:{lsn:2818400256}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:86016}, readable_log_tail:{lsn:86016}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2952609792}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/meta", dir_fd:125, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67018752, block_header_serialize_buf_:0x7f1ab0f4ebb8, flashback_version:0}, log_storage:{log_tail:{lsn:2944224980}, readable_log_tail:{lsn:2944224980}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:43, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:0, max_block_id:44, curr_writable_block_id:43}, logical_block_size_:67104768, curr_block_writable_size_:8384812, block_header_serialize_buf_:0x7f1ab0f508b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:-1, this:0x7f1ab0f4d0f0}, [Reconfirm]={palf_id:1, self:"127.0.0.1:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f1ab0f4cb30}) [2025-05-15 09:40:28.717604] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=236][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:28.717617] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", ack_info_list=[]) [2025-05-15 09:40:28.717627] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:28.717636] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] current log_task status(palf_id=1, self="127.0.0.1:2882", log_id=6576040, log_task={header:{begin_lsn:{lsn:2944224391}, end_lsn:{lsn:2944224980}, log_id:6576040, min_scn:{val:1747289697927760000, v:0}, max_scn:{val:1747289697927760000, v:0}, data_len:533, proposal_id:4, prev_lsn:{lsn:2944224269}, prev_proposal_id:4, committed_end_lsn:{lsn:2944224391}, data_checksum:1008861627, accum_checksum:3689732321, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1747300517815094, freeze_ts:1747300517815094, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:-1747300517815095, submit_to_flush cost time:0}, ack_info_list=[]) [2025-05-15 09:40:28.717669] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:396) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4018] fail to get server region(ret=-4018, server="127.0.0.1:2882") [2025-05-15 09:40:28.717680] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] get_server_region failed(server="127.0.0.1:2882") [2025-05-15 09:40:28.717691] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:398) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:28.717704] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2766) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:28.721918] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12330994037, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.721949] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.721976] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:28.721998] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.722011] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.722029] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.722042] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.722050] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028721886) [2025-05-15 09:40:28.722163] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.722193] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028722156) [2025-05-15 09:40:28.722213] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302028522101, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.722241] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12330821544, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.722259] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.722279] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:28.722295] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.722308] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.722324] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.722336] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.722345] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028722228) [2025-05-15 09:40:28.724813] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.731339] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try inc thread count(*this={name:TimerWK, this:0x7f1a8f8b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:6041207045, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:28.731433] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26] recycle one thread(this=0x7f1a8f8b0e70, total=5, remain=4) [2025-05-15 09:40:28.731452] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] inc thread count(*this={name:TimerWK, this:0x7f1a8f8b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:6041207045, tenant_id:1002}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:28.734922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.740969] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3954][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=27] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:28.745030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.752339] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.752360] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:28.752370] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:28.752389] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:28.752405] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:28.752436] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:28.755138] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.761010] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:28, tid:4602}]) [2025-05-15 09:40:28.764136] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.764161] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15507562, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.764179] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15507562, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.764207] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.764223] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.764232] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.764250] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.764268] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028764266, replica_locations:[]}) [2025-05-15 09:40:28.764288] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.764314] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.765246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.765466] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.765500] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.765517] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.765536] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.765551] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.765565] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.765581] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.765597] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028765596, replica_locations:[]}) [2025-05-15 09:40:28.765659] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15506022, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=192, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.775357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.785461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.795573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.799410] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.800166] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.800190] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028800160) [2025-05-15 09:40:28.800203] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302028600125, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.800232] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12331048905, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.800249] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.800268] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.800281] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.800289] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.800302] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12331048905, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.800337] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.800353] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.800362] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:28.800372] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:28.800393] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.800404] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.800411] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028800215) [2025-05-15 09:40:28.801657] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4486][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=20] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:28.805684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.812197] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.812219] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.812236] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.812255] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.812270] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.812286] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.812300] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.812309] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.812322] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302028812176, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.812338] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.812346] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.812409] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.812422] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.812429] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.812441] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.815792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.817351] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10] LogLoopThread round_cost_time(us)(round_cost_time=429) [2025-05-15 09:40:28.817375] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.817389] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.817401] WDIAG [PALF] get_leader_from_cache_ (log_sliding_window.cpp:2149) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock_get_leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:28.817412] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_fetch_log_dst failed or invalid(ret=-4721, palf_id=1, self="127.0.0.1:2882", fetch_log_dst="0.0.0.0:0") [2025-05-15 09:40:28.817423] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.819537] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=26] gc stale ls task succ [2025-05-15 09:40:28.820239] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.821948] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.821976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028821943) [2025-05-15 09:40:28.821995] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302028621897, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.822020] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12331093417, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.822039] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.822061] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:28.822079] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.822088] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.822104] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.822116] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.822123] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028822007) [2025-05-15 09:40:28.822234] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12330920924, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.822257] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.822279] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:28.822295] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.822303] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.822319] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.822330] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.822340] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028822222) [2025-05-15 09:40:28.825897] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.826716] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:28.826784] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4460][T1001_TimerWK0_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=16] ====== check clog disk timer task ====== [2025-05-15 09:40:28.826804] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4460][T1001_TimerWK0_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=16] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:28.836001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.836941] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1747302028836938, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:28.842821] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302028842818, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-05-15 09:40:28.846107] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.853551] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=23] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:28.854147] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4243][T1_Occam][T1][YB427F000001-000635291B4C65C9-0-0] [lt=24] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.856211] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.865780] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.865805] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15405916, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.865819] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15405916, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.865834] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.865853] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.865870] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.865883] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.865901] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.865918] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028865917, replica_locations:[]}) [2025-05-15 09:40:28.865938] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.865964] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.866315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.867145] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.867168] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.867185] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.867209] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.867224] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.867238] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.867255] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.867270] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028867269, replica_locations:[]}) [2025-05-15 09:40:28.867332] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15404349, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=193, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.876426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.886531] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.893760] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=19] ====== tenant freeze timer task ====== [2025-05-15 09:40:28.896634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.899468] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:28.900225] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.900249] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028900220) [2025-05-15 09:40:28.900290] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302028800215, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.900310] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:28.900345] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12331148228, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.900363] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.900384] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.900402] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.900414] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.900433] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12331148228, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:28.900448] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.900463] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:28.900477] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:28.900484] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:28.900505] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.900518] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:28.900525] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028900327) [2025-05-15 09:40:28.900556] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=44] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:28.900556] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=41] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:28.902003] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:28.906739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.912276] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.912299] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.912319] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.912339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.912354] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.912368] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.912376] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.912390] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302028912250, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.912407] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.912415] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:28.912481] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:28.912503] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:28.912511] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:28.912524] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:28.916846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.917434] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:28.917455] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:28.917467] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:28.922024] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12331193769, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:28.922052] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.922085] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:28.922100] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.922114] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.922125] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.922136] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:28.922143] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028922006) [2025-05-15 09:40:28.922278] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:28.922313] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302028922273) [2025-05-15 09:40:28.922333] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302028722228, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:28.922359] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12331021276, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:28.922376] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:28.922395] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:28.922410] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:28.922418] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:28.922433] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.922444] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:28.922452] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302028922349) [2025-05-15 09:40:28.926954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.931535] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4805677904, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:28.931684] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=27] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:28.931700] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4805677904, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:28.933413] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.933435] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.933454] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.933470] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.933497] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.933513] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.933522] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.933535] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.933549] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.933565] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:28.933574] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:28.933604] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=29][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:28.933625] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.933638] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.933652] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.933672] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.933686] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.933696] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.933710] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.933721] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028933720, replica_locations:[]}) [2025-05-15 09:40:28.933742] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:28.933785] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:28.933801] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.933838] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=37] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:28.933861] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=23][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:28.933885] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=24] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=118, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:28.933909] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=24][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:28.933936] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=25][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:28.933944] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:28.937063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.947174] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.949514] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [4648][T1002_ArcSender][T1002][YB427F000001-000635291E8C6112-0-0] [lt=21] ObArchiveSender is running(thread_index=3) [2025-05-15 09:40:28.951321] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=1] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:28.951360] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:28.954961] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:28.957284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.957574] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4531][T1002_Occam][T1002][YB427F000001-000635291EDC65C5-0-0] [lt=24] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.957799] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=31] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:28.957826] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-05-15 09:40:28.957840] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:28.957859] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:28.957877] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11791848220, flush_io_finished_ret_:0, flush_io_finished_round_:1503, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:28.961411] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:33, tid:3961}]) [2025-05-15 09:40:28.967389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.967451] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.967474] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15304243, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.967496] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15304243, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.967512] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.967520] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=15304243, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.967542] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.967560] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.967576] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.967591] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.967604] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.967621] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.967632] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028967631, replica_locations:[]}) [2025-05-15 09:40:28.967652] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:28.967679] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:28.967691] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:28.967721] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.967774] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792858251, cache_obj->added_lc()=false, cache_obj->get_object_id()=15716, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.968978] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:28.969000] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.969016] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:28.969032] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:28.969039] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:28.969054] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:28.969066] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:28.969082] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.969096] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:28.969109] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:28.969122] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:28.969136] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302028969136, replica_locations:[]}) [2025-05-15 09:40:28.969200] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15302481, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=194, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:28.977496] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.987603] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.988232] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221552698, support_ssl=false) [2025-05-15 09:40:28.988314] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=38] sm conn init succ(conn.sessid_=3221552698, sess.client_addr_="127.0.0.1:45086") [2025-05-15 09:40:28.988336] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552698, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.988509] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:28.988550] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:28.988579] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=34][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:28.988608] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:28.988628] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:28.988636] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:28.988669] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=19] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.988690] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552698, proxy_sessid=0) [2025-05-15 09:40:28.988724] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=16][errcode=0] server close connection(sessid=3221552698, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:28.988740] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552698, proxy_sessid=0) [2025-05-15 09:40:28.988748] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:28.988763] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9035-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552698, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=3908101, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:28.988811] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552698, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028988201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:118, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:28.988834] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552698, proxy_sessid=0, ret=0) [2025-05-15 09:40:28.988843] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552698, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302028988201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:118, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:28.988862] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552698) [2025-05-15 09:40:28.988880] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552698, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:28.995190] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD0-0-0] [lt=6][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.995211] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD0-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.995227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD0-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.995246] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD0-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.995260] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD0-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.995272] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:28.995287] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.995295] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.995308] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302028995166, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:28.995324] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.995334] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:28.995410] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:28.995939] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.995966] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:28.995975] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.995988] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:28.996004] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:28.996015] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:28.996033] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:28.996050] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:28.996087] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:28.996137] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:28.996659] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD1-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.996677] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD1-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:28.996688] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD1-0-0] [lt=7][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:28.996703] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD1-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:28.996711] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD1-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:28.996724] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:28.996732] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:28.996743] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:28.996750] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302028996650, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:28.996764] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:28.996772] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:28.996811] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:28.997711] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:28.998202] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=18] gc stale ls task succ [2025-05-15 09:40:28.999521] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.000303] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12331248603, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.000327] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.000355] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.000373] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.000387] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.000405] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12331248603, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.000420] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.000437] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.000450] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.000458] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.000472] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.000484] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.000502] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029000280) [2025-05-15 09:40:29.007818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.012384] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.012408] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.012427] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.012445] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.012460] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.012549] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=88][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.012563] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.012571] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.012585] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302029012356, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.012599] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.012607] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.012674] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.012686] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.012694] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.012706] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.017499] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.017523] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.017533] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.017928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.022067] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.022103] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029022061) [2025-05-15 09:40:29.022123] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302028822007, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.022162] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12331294127, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.022181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.022207] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.022222] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.022235] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.022247] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.022259] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.022269] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029022146) [2025-05-15 09:40:29.022343] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12331121634, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.022364] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.022384] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.022400] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.022409] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.022424] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.022436] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.022446] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029022330) [2025-05-15 09:40:29.026807] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:29.028034] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.038144] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.048061] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.048064] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC9666-0-0] [lt=19][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302029048046}) [2025-05-15 09:40:29.048096] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:29.048119] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:29.048138] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.048156] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.048251] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.048513] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:29.048527] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:29.048541] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.048594] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=52] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=9161, get_gts_with_stc_cnt=100, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:29.048610] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.053763] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552702, support_ssl=false) [2025-05-15 09:40:29.053818] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221552702, sess.client_addr_="172.22.0.1:44662") [2025-05-15 09:40:29.053839] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552702, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.053960] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=47] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.053995] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.054027] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.054061] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=33][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.054074] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.054089] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.054110] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.054130] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552702, proxy_sessid=0) [2025-05-15 09:40:29.054163] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=16][errcode=0] server close connection(sessid=3221552702, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.054178] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552702, proxy_sessid=0) [2025-05-15 09:40:29.054186] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.054194] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9043-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552702, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.054249] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552702, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029053701, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.054279] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221552702, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.054296] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552702, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029053701, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.054315] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552702) [2025-05-15 09:40:29.054331] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552702, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.058358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.067808] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:29.067839] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.067854] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:29.067863] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:29.067880] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:29.067960] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=80][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:29.068476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.069332] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.069359] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15202369, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.069376] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15202369, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.069393] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.069408] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=15202369, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.069426] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.069444] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.069462] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.069478] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.069501] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.069519] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.069537] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029069535, replica_locations:[]}) [2025-05-15 09:40:29.069556] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.069582] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.069595] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:29.069641] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.069692] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11792960168, cache_obj->added_lc()=false, cache_obj->get_object_id()=15717, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.070911] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.070932] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.070949] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.070965] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.070973] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.070988] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.071004] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.071020] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.071035] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.071043] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.071060] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.071076] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029071075, replica_locations:[]}) [2025-05-15 09:40:29.071142] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15200540, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=195, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.078589] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.088695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.089942] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552703, support_ssl=false) [2025-05-15 09:40:29.089986] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552703, sess.client_addr_="172.22.0.1:44674") [2025-05-15 09:40:29.090004] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552703, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.090119] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.090150] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.090149] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552704, support_ssl=false) [2025-05-15 09:40:29.090180] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.090204] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.090217] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.090224] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.090249] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090263] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552703, proxy_sessid=0) [2025-05-15 09:40:29.090268] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552704, sess.client_addr_="172.22.0.1:44686") [2025-05-15 09:40:29.090300] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=15][errcode=0] server close connection(sessid=3221552703, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090290] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552704, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.090313] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552703, proxy_sessid=0) [2025-05-15 09:40:29.090306] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552705, support_ssl=false) [2025-05-15 09:40:29.090322] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.090332] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9074-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552703, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.090354] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221552705, sess.client_addr_="172.22.0.1:44692") [2025-05-15 09:40:29.090372] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552705, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.090380] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552703, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029089631, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.090405] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.090407] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552703, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.090424] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552703, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029089631, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.090446] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221552703) [2025-05-15 09:40:29.090452] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.090463] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552703, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.090497] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.090518] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.090533] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.090542] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.090556] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552706, support_ssl=false) [2025-05-15 09:40:29.090578] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090599] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552706, sess.client_addr_="172.22.0.1:44706") [2025-05-15 09:40:29.090608] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=24][errcode=-4018] get session fail(ret=-4018, sessid=3221552704, proxy_sessid=0) [2025-05-15 09:40:29.090624] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552706, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.090651] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=16][errcode=0] server close connection(sessid=3221552704, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090665] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552704, proxy_sessid=0) [2025-05-15 09:40:29.090673] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:29.090688] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.090690] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9036-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552704, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.090723] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.090734] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.090745] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.090759] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.090748] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552704, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029089631, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.090766] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.090769] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552704, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.090782] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090781] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552704, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029089631, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.090791] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552705, proxy_sessid=0) [2025-05-15 09:40:29.090797] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552704) [2025-05-15 09:40:29.090808] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=7][errcode=0] server close connection(sessid=3221552705, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090814] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.090819] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552705, proxy_sessid=0) [2025-05-15 09:40:29.090830] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:29.090818] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552704, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.090840] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.090836] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9037-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552705, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.090856] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.090871] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.090881] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.090888] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.090884] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552705, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302029089631, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.090903] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090907] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552705, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.090916] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552706, proxy_sessid=0) [2025-05-15 09:40:29.090918] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552705, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302029089631, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.090935] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=9][errcode=0] server close connection(sessid=3221552706, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.090939] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552705) [2025-05-15 09:40:29.090945] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552706, proxy_sessid=0) [2025-05-15 09:40:29.090957] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:29.090955] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552705, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.090967] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9075-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552706, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.091009] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552706, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302029090691, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.091033] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552706, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.091068] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552706, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302029090691, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.091088] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552706) [2025-05-15 09:40:29.091100] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552706, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.091968] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552707, support_ssl=false) [2025-05-15 09:40:29.091970] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=41] new mysql sessid created(conn.sessid_=3221552708, support_ssl=false) [2025-05-15 09:40:29.092014] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552707, sess.client_addr_="172.22.0.1:44714") [2025-05-15 09:40:29.092023] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221552708, sess.client_addr_="172.22.0.1:44712") [2025-05-15 09:40:29.092030] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552707, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.092039] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552708, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.092096] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552709, support_ssl=false) [2025-05-15 09:40:29.092173] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=46] sm conn init succ(conn.sessid_=3221552709, sess.client_addr_="172.22.0.1:44734") [2025-05-15 09:40:29.092193] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552709, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.092262] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.092258] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.092292] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.092304] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.092321] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.092334] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.092347] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.092334] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.092354] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.092368] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.092364] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=26][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.092381] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552708, proxy_sessid=0) [2025-05-15 09:40:29.092383] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.092394] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.092399] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=8][errcode=0] server close connection(sessid=3221552708, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.092401] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.092410] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552708, proxy_sessid=0) [2025-05-15 09:40:29.092418] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.092426] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.092426] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9038-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552708, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.092444] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.092450] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=20][errcode=-4018] get session fail(ret=-4018, sessid=3221552707, proxy_sessid=0) [2025-05-15 09:40:29.092455] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.092467] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.092479] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.092470] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552708, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029091752, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.092495] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.092486] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=16][errcode=0] server close connection(sessid=3221552707, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.092511] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.092510] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=39] kill and revert session(conn.sessid_=3221552708, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.092521] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552709, proxy_sessid=0) [2025-05-15 09:40:29.092515] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=29][errcode=-4018] get session fail(ret=-4018, sessid=3221552707, proxy_sessid=0) [2025-05-15 09:40:29.092521] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552708, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029091752, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.092531] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:29.092537] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552708) [2025-05-15 09:40:29.092538] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=8][errcode=0] server close connection(sessid=3221552709, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.092550] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552709, proxy_sessid=0) [2025-05-15 09:40:29.092557] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.092553] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552708, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.092549] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9044-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552707, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.092564] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9039-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552709, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.092614] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552709, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029091752, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.092623] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552707, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302029091752, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.092640] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552709, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.092646] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552707, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.092649] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552709, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029091752, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.092661] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552707, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302029091752, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.092667] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552709) [2025-05-15 09:40:29.092680] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552707) [2025-05-15 09:40:29.092694] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552707, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.092690] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552709, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.094591] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552710, support_ssl=false) [2025-05-15 09:40:29.094641] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552710, sess.client_addr_="172.22.0.1:44742") [2025-05-15 09:40:29.094660] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552710, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.094791] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.094826] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.094844] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.094859] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.094869] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.094879] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.094893] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.094905] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552710, proxy_sessid=0) [2025-05-15 09:40:29.094931] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=15][errcode=0] server close connection(sessid=3221552710, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.094940] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552710, proxy_sessid=0) [2025-05-15 09:40:29.094949] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:29.094958] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9076-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552710, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.095003] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552710, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029093878, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.095029] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552710, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.095037] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552710, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029093878, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.095057] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552710) [2025-05-15 09:40:29.095068] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552710, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.095673] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221552711, support_ssl=false) [2025-05-15 09:40:29.095712] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552711, sess.client_addr_="172.22.0.1:44754") [2025-05-15 09:40:29.095729] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552711, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.095752] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552712, support_ssl=false) [2025-05-15 09:40:29.095789] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552712, sess.client_addr_="172.22.0.1:44760") [2025-05-15 09:40:29.095806] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552712, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.095805] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552713, support_ssl=false) [2025-05-15 09:40:29.095855] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552713, sess.client_addr_="172.22.0.1:44772") [2025-05-15 09:40:29.095873] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552713, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.095930] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.095962] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.095982] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.096001] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.096016] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.096016] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.096024] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.096041] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.096050] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.096055] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552711, proxy_sessid=0) [2025-05-15 09:40:29.096075] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.096079] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=11][errcode=0] server close connection(sessid=3221552711, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.096072] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=32][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.096094] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552711, proxy_sessid=0) [2025-05-15 09:40:29.096096] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.096102] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:29.096106] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.096113] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.096114] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.096108] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903A-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552711, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.096129] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.096129] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.096141] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552712, proxy_sessid=0) [2025-05-15 09:40:29.096143] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.096152] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.096159] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.096153] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552711, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302029094938, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.096162] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=8][errcode=0] server close connection(sessid=3221552712, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.096175] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552711, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.096177] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552712, proxy_sessid=0) [2025-05-15 09:40:29.096188] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:29.096184] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552711, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302029094938, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.096197] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221552711) [2025-05-15 09:40:29.096201] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=38] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.096213] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552713, proxy_sessid=0) [2025-05-15 09:40:29.096196] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9045-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552712, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.096213] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552711, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.096248] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=24][errcode=0] server close connection(sessid=3221552713, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.096262] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552713, proxy_sessid=0) [2025-05-15 09:40:29.096269] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:29.096264] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552712, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029096001, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.096286] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552712, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.096280] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9077-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552713, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.096294] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552712, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029096001, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.096313] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552712) [2025-05-15 09:40:29.096326] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552712, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.096329] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552713, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029096001, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.096357] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552713, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.096369] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552713, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029096001, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.096384] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552713) [2025-05-15 09:40:29.096399] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552713, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.098803] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.099587] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.100346] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.100391] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029100339) [2025-05-15 09:40:29.100421] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302028900300, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.100452] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12331349039, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.100472] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.100503] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.100521] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.100534] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.100550] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12331349039, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.100565] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.100579] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.100591] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.100599] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.100611] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.100622] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.100633] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029100436) [2025-05-15 09:40:29.102284] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:29.108922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.112444] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.112467] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.112503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.112524] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.112542] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.112555] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.112571] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.112586] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.112600] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302029112414, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.112617] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.112632] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.112710] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.112724] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.112738] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.112751] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.114350] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:29.114369] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:29.114380] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=10] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:29.114397] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:29.117532] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.117558] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.117569] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.119036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.119747] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=17] gc stale ls task succ [2025-05-15 09:40:29.122147] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12331393552, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.122172] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.122196] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.122213] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.122225] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.122240] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.122250] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.122260] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029122123) [2025-05-15 09:40:29.122395] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.122420] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029122388) [2025-05-15 09:40:29.122437] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302028922349, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.122467] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12331222119, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.122481] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.122512] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.122524] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.122534] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.122548] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.122558] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.122569] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029122456) [2025-05-15 09:40:29.127045] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:29.127072] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-05-15 09:40:29.127085] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:29.127101] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:29.127117] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11792017459, flush_io_finished_ret_:0, flush_io_finished_round_:1503, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:29.129147] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.139257] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.142905] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4289][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=6] ====== check clog disk timer task ====== [2025-05-15 09:40:29.142928] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4289][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:29.149352] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.159466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.166594] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4462][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=21] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:29.169588] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.171269] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.171294] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15100430, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.171311] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=15100430, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.171325] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.171338] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=15100430, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.171353] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.171370] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.171386] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.171401] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.171412] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.171428] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.171444] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029171442, replica_locations:[]}) [2025-05-15 09:40:29.171463] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.171499] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.171511] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:29.171539] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.171592] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11793062067, cache_obj->added_lc()=false, cache_obj->get_object_id()=15718, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.172246] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=15] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:29.172276] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:29.172292] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:29.172306] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:29.172320] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:29.172331] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:29.172346] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:29.172358] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:29.172367] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:29.172376] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:29.172387] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:29.172395] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:29.172405] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:29.172414] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:29.172436] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:29.172447] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.172459] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.172470] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:29.172481] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:29.172504] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=20][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:29.172514] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:29.172533] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:29.172556] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:29.172565] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:29.172573] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:29.172589] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:29.172601] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646A-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.172612] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:29.172625] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.172636] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:29.172646] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:29.172655] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca3b0, start=1747302029171979, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.172667] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:29.172677] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.172756] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=12][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:29.172769] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=12][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:29.172780] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=10][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:29.172790] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:29.172790] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.172802] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=9][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:29.172816] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=10][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:29.172823] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646A-0-0] [lt=7][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:29.172819] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.172842] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.172858] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.172866] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.172880] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.172893] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.172907] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.172921] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.172932] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.172948] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.172962] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029172961, replica_locations:[]}) [2025-05-15 09:40:29.173025] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=15098657, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=196, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.173175] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.173299] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.173323] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.174057] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.179697] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.189806] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.190798] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=21] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:29.199640] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.200262] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.200417] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12331448474, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.200439] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.200470] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.200496] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.200611] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=115][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.200634] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=12331448474, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.200648] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.200660] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.200672] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.200679] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.200692] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.200702] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.200712] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029200400) [2025-05-15 09:40:29.210901] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.212382] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.212404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.212420] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.212436] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.212447] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.212460] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.212469] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.212476] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.212493] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302029212364, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.212506] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.212514] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.212567] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.212578] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.212589] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.212599] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.213560] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=22][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:29.213582] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.213594] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.213611] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.213625] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.213641] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.213655] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.213668] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.213682] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.213695] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.213707] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.213723] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.213749] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.213765] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:29.215348] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:29.215376] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:29.215395] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:29.215412] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:29.215426] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:29.215441] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:29.215453] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:29.215465] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:29.215476] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:29.215486] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:29.215507] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=20][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:29.215520] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:29.215530] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:29.215540] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:29.215559] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:29.215570] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.215583] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.215594] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:29.215604] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:29.215616] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:29.215626] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:29.215644] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:29.215665] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:29.215672] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:29.215681] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:29.215698] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:29.215712] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.215724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:29.215736] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.215747] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:29.215758] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:29.215769] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302029215157, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.215783] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:29.215791] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.215872] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:29.215886] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=13][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:29.215898] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=12][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:29.215906] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=7][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:29.215917] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:29.215929] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=11][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:29.215940] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638E-0-0] [lt=10][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:29.217591] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.217614] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.217625] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.221015] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.222187] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:29.222209] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.222234] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029222181) [2025-05-15 09:40:29.222254] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302029022138, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.222293] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12331494015, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.222313] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.222342] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.222361] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.222375] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.222394] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.222406] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.222422] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029222271) [2025-05-15 09:40:29.222456] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12331321522, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.222475] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.222504] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.222516] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.222525] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.222537] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.222544] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.222550] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029222445) [2025-05-15 09:40:29.226865] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=6) [2025-05-15 09:40:29.231126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.241232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.251340] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.252507] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.252531] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:29.252543] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:29.252555] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:29.252569] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:29.252625] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=53] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:29.257502] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=51] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:29.257531] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-05-15 09:40:29.257545] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:29.257562] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:29.257576] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11792147911, flush_io_finished_ret_:0, flush_io_finished_round_:1503, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:29.259439] WDIAG [OCCAM] dump_statistics (ob_vtable_event_recycle_buffer.h:144) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65C6-0-0] [lt=19][errcode=-4006] not init(ret=-4006) [2025-05-15 09:40:29.259464] INFO [MDS] dump_map_holding_item (mds_tenant_service.cpp:462) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65C6-0-0] [lt=18] finish scan map holding items(scan_cnt=0) [2025-05-15 09:40:29.261450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.262023] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:339, tid:4304}]) [2025-05-15 09:40:29.262058] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65C7-0-0] [lt=10] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.263571] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=24] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:29.271569] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.273175] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.273204] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14998528, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.273223] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14998528, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.273236] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.273247] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14998528, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.273262] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.273299] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=34][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.273316] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.273332] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.273344] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.273360] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.273380] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029273377, replica_locations:[]}) [2025-05-15 09:40:29.273408] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.273429] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:196, local_retry_times:196, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:29.273449] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.273461] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:29.273474] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.273484] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.273502] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:29.273535] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:29.273550] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.273601] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11793164077, cache_obj->added_lc()=false, cache_obj->get_object_id()=15719, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.274862] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.274890] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.274929] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.274945] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.274961] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.274974] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.274985] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.274999] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.275014] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.275024] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.275038] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.275050] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.275063] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.275078] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029275077, replica_locations:[]}) [2025-05-15 09:40:29.275094] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.275109] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.275122] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.275138] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:29.275151] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:29.275159] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:29.275177] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:29.275192] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.275202] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.275213] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:29.275225] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:29.275233] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:29.275246] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:29.275258] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:29.275270] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:29.275279] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:29.275290] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:29.275301] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:29.275312] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:29.275329] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:29.275341] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.275353] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:29.275364] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:29.275375] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:29.275385] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=197, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:29.275412] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=14996269, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=197, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.281681] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.291791] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.296186] WDIAG load_file_to_string (utility.h:673) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:29.296212] WDIAG get_ethernet_speed (utility.cpp:575) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:29.296224] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:29.296236] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:29.299698] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.300468] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.300501] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029300461) [2025-05-15 09:40:29.300602] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=101][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302029100435, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.300639] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12331548889, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.300658] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.300683] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.300701] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.300715] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.300732] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12331548889, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.300745] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.300760] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.300772] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.300782] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.300794] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.300804] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.300815] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029300616) [2025-05-15 09:40:29.301895] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.302559] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:29.312002] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.312594] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.312615] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.312640] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.312656] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.312668] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.312680] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.312692] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.312703] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.312713] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302029312560, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.312727] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.312737] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.312814] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.312825] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.312835] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.312845] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.317648] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.317669] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.317679] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.318709] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:29.318734] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.318752] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.318770] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:29.318782] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:29.318795] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:29.322113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.322269] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12331593380, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.322299] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.322327] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.322349] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.322364] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.322384] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.322397] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.322410] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029322242) [2025-05-15 09:40:29.322512] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.322538] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029322505) [2025-05-15 09:40:29.322561] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302029122455, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.322598] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12331421942, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.322615] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.322710] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=94][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.322731] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.322745] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.322767] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.322778] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.322790] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029322587) [2025-05-15 09:40:29.326897] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.326917] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.326929] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:29.326956] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:29.329554] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:29.329578] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.329594] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.332224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.342332] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.352106] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-05-15 09:40:29.352141] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.352154] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:29.352164] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:29.352175] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:29.352253] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=79][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:29.352265] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.352273] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:29.352282] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:29.352290] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:29.352339] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:29.352438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.354565] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:29.354642] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:29.354715] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:29.354779] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:29.354813] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:5060, total:5071) [2025-05-15 09:40:29.354850] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:29.354901] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:5060, total:5071) [2025-05-15 09:40:29.354932] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:29.354953] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:29.354968] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:29.354993] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] successfully alloc block(macro_id={[ver=1,mode=0,seq=628764][2nd=1891]}) [2025-05-15 09:40:29.362545] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.372653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.375543] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.375567] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14896157, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.375583] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14896157, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.375597] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.375606] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14896157, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.375617] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.375633] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.375648] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.375662] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.375673] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.375688] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.375709] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029375708, replica_locations:[]}) [2025-05-15 09:40:29.375727] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.375747] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:197, local_retry_times:197, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:29.375767] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.375779] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:29.375796] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.375805] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.375811] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:29.375830] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:29.375843] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.375890] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11793266367, cache_obj->added_lc()=false, cache_obj->get_object_id()=15720, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.376982] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.377009] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.377051] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.377065] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.377079] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.377094] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.377102] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.377115] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.377126] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.377140] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.377153] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.377164] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.377178] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.377192] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029377191, replica_locations:[]}) [2025-05-15 09:40:29.377208] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.377221] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.377233] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.377250] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:29.377257] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:29.377269] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:29.377287] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:29.377299] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.377310] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.377317] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:29.377328] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:29.377339] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:29.377351] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:29.377362] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:29.377371] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:29.377381] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:29.377388] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:29.377398] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:29.377406] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:29.377426] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:29.377433] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.377444] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:29.377456] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:29.377466] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:29.377476] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=198, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:29.377511] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24] will sleep(sleep_us=100000, remain_us=14894170, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=198, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.383743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.388888] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:29.388917] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2025-05-15 09:40:29.388952] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=34][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:29.388967] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2025-05-15 09:40:29.388981] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:29.388993] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:29.389010] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2025-05-15 09:40:29.389022] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:29.389063] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=41][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:29.389074] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:29.389082] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:6094) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=7][errcode=-5019] resolve table failed(ret=-5019) [2025-05-15 09:40:29.389090] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:5649) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=7][errcode=-5019] resolve joined table item failed(ret=-5019) [2025-05-15 09:40:29.389101] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5457) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10][errcode=-5019] resolve joined table failed(ret=-5019) [2025-05-15 09:40:29.389111] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:29.389122] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:29.389133] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:29.389145] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:29.389158] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=7][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:29.389169] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.389182] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.389190] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=7][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:29.389200] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2025-05-15 09:40:29.389208] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=8][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:29.389219] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:29.389236] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:29.389254] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:29.389264] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:29.389269] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:29.389286] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:29.389299] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4776][BlackListServic][T1][YB427F000001-000635291F1C6307-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.389311] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4776][BlackListServic][T0][YB427F000001-000635291F1C6307-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2025-05-15 09:40:29.389324] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:29.389336] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:29.389347] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:29.389358] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302029388603, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:29.389374] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:114) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:29.389383] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:29.389459] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12] ls blacklist refresh finish(cost_time=2254) [2025-05-15 09:40:29.393860] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.395423] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.395520] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=28] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.395802] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=22] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.396161] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.396274] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=24] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.396749] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.396938] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:29.399757] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.400556] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12331649203, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.400579] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.400617] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.400635] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.400643] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.400659] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12331649203, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.400673] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.400688] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.400700] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.400710] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.400721] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.400732] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.400740] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029400534) [2025-05-15 09:40:29.403972] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.411910] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:29.411943] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.411963] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:29.411980] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:29.412624] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.412641] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.412660] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.412675] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.412687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.412698] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.412709] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.412720] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.412730] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302029412596, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.412743] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.412752] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.412816] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.412828] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.412838] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.412845] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.414082] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.417714] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.417738] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.417748] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.418768] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:29.418791] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.418807] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.418824] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:29.418836] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:29.418848] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:29.422306] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.422335] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029422299) [2025-05-15 09:40:29.422355] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302029222270, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.422395] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12331693715, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.422415] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.422440] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.422461] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.422475] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.422506] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.422519] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.422533] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029422375) [2025-05-15 09:40:29.422577] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12331521222, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.422596] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.422613] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.422627] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.422635] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.422647] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.422655] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.422663] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029422564) [2025-05-15 09:40:29.424194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.426957] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:29.429608] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:29.429627] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.429640] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.434315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.444424] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.454529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.458132] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:29.458168] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:29.464641] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.474762] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.477654] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.477687] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14794047, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.477704] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14794047, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.477719] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.477731] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14794047, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.477746] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.477832] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=84][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.477848] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.477863] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.477872] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.477888] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.477907] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029477905, replica_locations:[]}) [2025-05-15 09:40:29.477926] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.477942] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:198, local_retry_times:198, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:29.477963] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.477976] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:29.477997] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.478006] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.478012] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:29.478045] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:29.478058] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.478101] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11793368578, cache_obj->added_lc()=false, cache_obj->get_object_id()=15721, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.479247] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.479278] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=30][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.479319] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.479339] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.479354] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.479365] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.479378] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.479392] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.479407] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.479421] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.479435] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.479446] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.479461] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.479475] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029479474, replica_locations:[]}) [2025-05-15 09:40:29.479505] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.479515] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.479524] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.479546] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:29.479562] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:29.479576] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:29.479596] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:29.479611] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.479618] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.479630] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:29.479639] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:29.479650] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:29.479663] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:29.479675] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:29.479682] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:29.479693] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:29.479704] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:29.479711] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:29.479722] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:29.479739] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:29.479751] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.479762] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:29.479774] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:29.479786] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:29.479795] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=199, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:29.479818] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=14791863, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=199, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.484871] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.494983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.496143] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:29.496177] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:29.496192] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:29.496205] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:29.496213] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:29.496223] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:29.496241] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:29.496255] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:29.496289] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:29.496297] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:29.496319] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.496333] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:29.496347] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:29.496360] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:29.496371] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:29.496384] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:29.496393] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:29.499834] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.500593] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.500615] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029500586) [2025-05-15 09:40:29.500630] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302029300615, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.500668] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12331748594, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.500686] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.500707] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.500725] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.500737] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.500753] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12331748594, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.500766] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.500780] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.500792] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.500799] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.500810] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.500820] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.500827] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029500644) [2025-05-15 09:40:29.502863] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:29.505080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.512748] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.512770] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.512791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.512810] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.512821] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.512834] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.512846] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.512858] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.512867] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302029512716, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.512879] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.512889] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.512968] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.512979] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.512988] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.512999] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.515188] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.517812] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.517859] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.517896] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.518848] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:29.518870] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.518887] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.518907] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:29.518918] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:29.518927] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:29.522403] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12331794157, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.522433] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.522456] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.522475] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.522628] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.522666] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029522622) [2025-05-15 09:40:29.522685] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302029322577, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.522720] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12331621664, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.522740] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.522761] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.525297] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.529667] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:29.529690] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.529705] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.535406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.536979] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=24] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:29.541502] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=20] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:29.547732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.549021] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.549043] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.549075] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:29.549088] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:29.549109] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.549125] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.549141] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] refresh gts failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:29.549153] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:29.549167] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:29.549181] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.549208] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:29.549221] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:29.549236] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:29.549249] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:29.549263] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:29.549276] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:29.549293] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.549897] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:29.549914] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.549943] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:29.549953] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:29.549966] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:29.549978] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:29.549990] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:29.550002] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:29.550021] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:29.554926] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552721, support_ssl=false) [2025-05-15 09:40:29.555042] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=46] sm conn init succ(conn.sessid_=3221552721, sess.client_addr_="172.22.0.1:44786") [2025-05-15 09:40:29.555063] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552721, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.555256] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.555296] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.555396] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.555439] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=39][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.555456] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.555466] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.555507] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.555529] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552721, proxy_sessid=0) [2025-05-15 09:40:29.555570] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=16][errcode=0] server close connection(sessid=3221552721, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.555585] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552721, proxy_sessid=0) [2025-05-15 09:40:29.555593] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.555603] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903B-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552721, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.555658] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552721, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029554685, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.555683] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552721, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.555699] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552721, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029554685, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.555718] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552721) [2025-05-15 09:40:29.555737] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552721, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.557851] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.559708] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6670-0-0] [lt=16][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.559733] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6670-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.559755] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6670-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.559774] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6670-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.559787] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.559804] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.559820] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=15][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.559834] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.559848] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302029559691, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.559863] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.559878] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:29.559942] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=14][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:29.559958] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=14][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:29.559973] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=15][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:29.559986] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=12][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:29.560002] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=12][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:29.560018] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=14][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:29.560030] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6670-0-0] [lt=12][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:29.567966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.573667] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4336][T1_ArcSender][T1][YB427F000001-000635291B9C6112-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:29.578077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.579949] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.579977] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14691753, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.579995] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14691753, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.580009] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.580022] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14691753, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.580038] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.580055] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.580073] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.580089] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.580102] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.580120] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.580139] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029580137, replica_locations:[]}) [2025-05-15 09:40:29.580158] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.580180] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:199, local_retry_times:199, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:29.580207] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.580220] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:29.580237] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.580248] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:29.580255] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:29.580276] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:29.580291] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.580350] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11793470824, cache_obj->added_lc()=false, cache_obj->get_object_id()=15722, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.581663] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.581692] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.581731] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.581745] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.581760] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.581774] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.581787] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.581801] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.581817] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.581831] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.581845] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.581856] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.581870] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.581885] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029581884, replica_locations:[]}) [2025-05-15 09:40:29.581902] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.581915] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.581924] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:29.581940] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:29.581953] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:29.581964] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:29.581983] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:29.581998] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.582009] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:29.582020] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:29.582032] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:29.582043] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:29.582056] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:29.582069] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:29.582080] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:29.582087] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:29.582098] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:29.582110] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:29.582121] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:29.582138] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:29.582150] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:29.582158] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:29.582170] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:29.582184] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:29.582195] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=200, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:29.582223] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=14689459, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=200, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.588198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.591474] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552723, support_ssl=false) [2025-05-15 09:40:29.591537] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552724, support_ssl=false) [2025-05-15 09:40:29.591561] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=55] sm conn init succ(conn.sessid_=3221552723, sess.client_addr_="172.22.0.1:44800") [2025-05-15 09:40:29.591581] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552723, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.591720] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552725, support_ssl=false) [2025-05-15 09:40:29.591739] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=84] sm conn init succ(conn.sessid_=3221552724, sess.client_addr_="172.22.0.1:44810") [2025-05-15 09:40:29.591757] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552725, sess.client_addr_="172.22.0.1:44824") [2025-05-15 09:40:29.591762] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552724, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.591775] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552725, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.591819] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.591868] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.591931] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.591927] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552726, support_ssl=false) [2025-05-15 09:40:29.591943] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.591963] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.591973] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=36][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.591989] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.591997] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.592002] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221552726, sess.client_addr_="172.22.0.1:44826") [2025-05-15 09:40:29.592010] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.592021] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.592013] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.592017] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552726, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.592035] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.592047] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592056] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.592065] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.592067] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552723, proxy_sessid=0) [2025-05-15 09:40:29.592085] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592104] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552725, proxy_sessid=0) [2025-05-15 09:40:29.592073] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.592111] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=15][errcode=0] server close connection(sessid=3221552723, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592126] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552723, proxy_sessid=0) [2025-05-15 09:40:29.592121] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=44][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.592130] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=10][errcode=0] server close connection(sessid=3221552725, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592143] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:29.592144] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552725, proxy_sessid=0) [2025-05-15 09:40:29.592144] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=22][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.592152] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.592156] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.592155] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9046-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552723, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.592161] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903C-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552725, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.592181] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.592212] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.592222] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.592230] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=8][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.592239] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.592246] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.592227] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552725, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.592288] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=61] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592310] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552724, proxy_sessid=0) [2025-05-15 09:40:29.592288] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552723, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.592326] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=77] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592330] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=37] kill and revert session(conn.sessid_=3221552725, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.592336] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552726, proxy_sessid=0) [2025-05-15 09:40:29.592341] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=12][errcode=0] server close connection(sessid=3221552724, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592351] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552724, proxy_sessid=0) [2025-05-15 09:40:29.592354] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=9][errcode=0] server close connection(sessid=3221552726, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.592346] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221552723, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.592367] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552726, proxy_sessid=0) [2025-05-15 09:40:29.592365] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:29.592374] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:29.592371] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552725, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.592381] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903D-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552726, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.592379] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9078-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552724, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.592403] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] mark session id unused(conn.sessid_=3221552725) [2025-05-15 09:40:29.592426] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552726, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.592451] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552726, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.592430] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552725, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.592445] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552724, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.592459] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552726, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.592471] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552724, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.592478] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552726) [2025-05-15 09:40:29.592465] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=33] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552723, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.592482] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552724, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302029591604, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.592499] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=34] mark session id unused(conn.sessid_=3221552723) [2025-05-15 09:40:29.592508] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] mark session id unused(conn.sessid_=3221552724) [2025-05-15 09:40:29.592506] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552726, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.592518] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552723, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.592524] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552724, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.593562] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=31] new mysql sessid created(conn.sessid_=3221552727, support_ssl=false) [2025-05-15 09:40:29.593608] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552727, sess.client_addr_="172.22.0.1:44838") [2025-05-15 09:40:29.593627] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552727, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.593656] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552728, support_ssl=false) [2025-05-15 09:40:29.593680] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221552728, sess.client_addr_="172.22.0.1:44848") [2025-05-15 09:40:29.593695] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221552728, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.593757] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552729, support_ssl=false) [2025-05-15 09:40:29.593788] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552729, sess.client_addr_="172.22.0.1:44858") [2025-05-15 09:40:29.593804] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de2adb0, session_id:3221552729, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.593841] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.593870] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.593887] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.593891] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.593906] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.593916] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.593919] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.593931] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.593937] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.593945] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.593940] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.593956] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.593957] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552727, proxy_sessid=0) [2025-05-15 09:40:29.593962] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.593973] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.593979] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=11][errcode=0] server close connection(sessid=3221552727, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.593977] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.593991] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552727, proxy_sessid=0) [2025-05-15 09:40:29.593993] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.593995] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=18][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.593998] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:29.594002] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.594011] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.594011] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.594007] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9047-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552727, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.594028] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552728, proxy_sessid=0) [2025-05-15 09:40:29.594033] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.594047] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552729, proxy_sessid=0) [2025-05-15 09:40:29.594050] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=8][errcode=0] server close connection(sessid=3221552728, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.594066] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552728, proxy_sessid=0) [2025-05-15 09:40:29.594064] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=7][errcode=0] server close connection(sessid=3221552729, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.594074] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.594063] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552727, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029593723, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.594076] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552729, proxy_sessid=0) [2025-05-15 09:40:29.594084] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552727, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.594089] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:29.594082] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903E-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552728, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.594096] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552727, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029593723, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.594101] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9079-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552729, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.594118] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552727) [2025-05-15 09:40:29.594130] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552727, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.594166] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221552728, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302029593723, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.594204] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2adb0, session_id:3221552729, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029593723, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.594228] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552728, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.594237] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221552729, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.594275] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221552728, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302029593723, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.594294] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552728) [2025-05-15 09:40:29.594306] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552728, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.594327] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f1a1de2adb0, session_id:3221552729, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302029593723, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.594339] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552729) [2025-05-15 09:40:29.594356] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552729, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.596033] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552730, support_ssl=false) [2025-05-15 09:40:29.596101] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] sm conn init succ(conn.sessid_=3221552730, sess.client_addr_="172.22.0.1:44872") [2025-05-15 09:40:29.596126] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552730, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.596302] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.596336] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.596350] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.596369] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.596376] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.596390] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.596405] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.596421] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552730, proxy_sessid=0) [2025-05-15 09:40:29.596439] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=8][errcode=0] server close connection(sessid=3221552730, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.596453] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552730, proxy_sessid=0) [2025-05-15 09:40:29.596462] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:29.596475] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9048-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552730, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.596535] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552730, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029595840, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.596562] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552730, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.596570] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552730, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302029595840, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.596589] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552730) [2025-05-15 09:40:29.596604] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552730, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.597394] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552731, support_ssl=false) [2025-05-15 09:40:29.597397] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=37] new mysql sessid created(conn.sessid_=3221552732, support_ssl=false) [2025-05-15 09:40:29.597425] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552733, support_ssl=false) [2025-05-15 09:40:29.597439] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552731, sess.client_addr_="172.22.0.1:44880") [2025-05-15 09:40:29.597445] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552732, sess.client_addr_="172.22.0.1:44892") [2025-05-15 09:40:29.597450] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552731, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.597459] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552732, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.597478] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552733, sess.client_addr_="172.22.0.1:44894") [2025-05-15 09:40:29.597510] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552733, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.597649] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=130] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.597674] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.597670] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:29.597684] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.597702] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.597706] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.597712] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=26] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:29.597727] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.597726] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=29][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.597739] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.597724] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:29.597742] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.597746] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.597750] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.597748] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:29.597758] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.597759] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:29.597762] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.597773] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:29.597776] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552732, proxy_sessid=0) [2025-05-15 09:40:29.597776] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.597790] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552733, proxy_sessid=0) [2025-05-15 09:40:29.597795] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.597800] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=11][errcode=0] server close connection(sessid=3221552732, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.597809] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552732, proxy_sessid=0) [2025-05-15 09:40:29.597810] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552731, proxy_sessid=0) [2025-05-15 09:40:29.597811] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=7][errcode=0] server close connection(sessid=3221552733, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.597817] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:29.597823] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552733, proxy_sessid=0) [2025-05-15 09:40:29.597825] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC903F-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552732, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.597838] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:29.597837] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=13][errcode=0] server close connection(sessid=3221552731, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.597848] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552731, proxy_sessid=0) [2025-05-15 09:40:29.597845] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9049-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552733, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.597858] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:29.597869] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907A-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552731, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:29.597887] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552732, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302029596900, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.597915] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552731, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029596900, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.597904] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552733, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302029596900, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:29.597922] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221552732, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.597935] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221552731, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.597939] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=32] kill and revert session(conn.sessid_=3221552733, proxy_sessid=0, ret=0) [2025-05-15 09:40:29.597942] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552732, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302029596900, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.597949] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552733, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302029596900, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.597945] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552731, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302029596900, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:29.597965] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552733) [2025-05-15 09:40:29.597965] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552731) [2025-05-15 09:40:29.597965] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552732) [2025-05-15 09:40:29.597978] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552733, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.597981] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552732, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.597982] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552731, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:29.598313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.599888] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.600677] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12331848882, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.600711] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.600743] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.600775] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12331848882, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.600791] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.608434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.609554] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4291][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=18] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4807739103, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:29.609959] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4291][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=45] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4807739115, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:29.612686] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.612709] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.612726] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.612746] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.612762] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.612773] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.612788] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.612797] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.612811] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302029612663, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.612827] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.612835] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.612897] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.612911] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.612919] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.612933] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.617844] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:29.617873] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:29.617886] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:29.618556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.622431] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.622471] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029622423) [2025-05-15 09:40:29.622500] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302029422374, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.622530] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12331894488, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.622551] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.622696] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12331721995, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.622717] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.627046] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:29.628680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.638796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.640642] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EABDC-0-0] [lt=27][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302029213744}) [2025-05-15 09:40:29.648907] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.660571] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.662837] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:32, tid:4630}]) [2025-05-15 09:40:29.670690] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.680805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.682392] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.682436] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=36][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14589325, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.682467] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=30][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14589325, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.682501] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=33][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.682527] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14589325, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.682568] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.682596] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.682647] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=49] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.682675] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.682708] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029682705, replica_locations:[]}) [2025-05-15 09:40:29.682741] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.682783] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.684565] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.684598] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.684623] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.684658] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.684681] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.684703] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.684722] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.684753] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029684752, replica_locations:[]}) [2025-05-15 09:40:29.684844] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14586838, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=201, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.690921] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.699938] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.700712] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.700757] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029700705) [2025-05-15 09:40:29.700778] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302029500643, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.700818] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12331949469, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.700843] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.700873] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.700894] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.700909] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.700924] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12331949469, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.700940] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.700957] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.700968] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.700981] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.700992] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.701007] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.701015] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029700795) [2025-05-15 09:40:29.701036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.703134] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:29.711145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.712785] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.712808] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.712826] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.712849] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.712864] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.712875] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.712889] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.712906] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302029712762, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.712922] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.712931] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.712998] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.713012] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.713020] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.713033] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.717897] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.718975] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.721260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.722512] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12331993997, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.722540] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.722564] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.722585] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.722595] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.722614] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.722658] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.722665] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029722484) [2025-05-15 09:40:29.722749] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.722774] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029722743) [2025-05-15 09:40:29.722794] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302029522698, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.722822] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12331821504, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.722842] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.722863] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.722881] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.722891] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.722909] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.722922] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.722930] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029722810) [2025-05-15 09:40:29.729778] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.731375] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.741064] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3953][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=24] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:29.741497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.751612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.752662] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.752689] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:29.752699] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:29.752715] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:29.752726] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:29.752762] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:29.761726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.771843] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.781957] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.785026] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.785070] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=36][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14486691, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.785097] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14486691, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.785138] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.785161] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.785178] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.785199] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.785225] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029785222, replica_locations:[]}) [2025-05-15 09:40:29.785250] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.785284] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.786800] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.786826] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.786845] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.786871] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.786888] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.786899] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.786917] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.786935] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029786934, replica_locations:[]}) [2025-05-15 09:40:29.787011] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14484670, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=202, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.791756] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [4292][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=23] BlockGCTimerTask success(ret=0, cost_time_us=11, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"127.0.0.1:2882", log_dir:"/root/ob/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-15 09:40:29.792086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.800015] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:29.800040] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:29.800052] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.800801] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12332048920, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.800828] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.800879] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.800902] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.800917] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.800931] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12332048920, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.800947] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.800960] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.800975] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.800983] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.800999] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.801013] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.801021] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029800775) [2025-05-15 09:40:29.802217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.812327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.812788] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.812811] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.812827] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.812846] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.819594] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=22] gc stale ls task succ [2025-05-15 09:40:29.822437] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.822544] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.822569] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029822539) [2025-05-15 09:40:29.822583] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302029622510, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.822654] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=59] get wrs ts(ls_id={id:1}, delta=12332094485, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.822673] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.822697] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.822716] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.822725] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.822742] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.822755] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.822762] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029822643) [2025-05-15 09:40:29.822817] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12331921992, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.822841] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.822864] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.822881] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.822889] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.822907] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.822920] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.822927] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029822806) [2025-05-15 09:40:29.827143] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:29.832178] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [5605][T1_TimerWK4_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.832558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.838846] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] [COMPACTION DAG STATUS] (start_time=1747302019838022, end_time=1747302029838844, dag_status={}) [2025-05-15 09:40:29.838872] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.838886] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.838960] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=74] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.838968] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.838976] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.838990] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.838998] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.839012] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.839022] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:29.839039] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-15 09:40:29.839053] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-15 09:40:29.842308] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4292][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=40] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:29.842667] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.844557] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302029844554, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:29.850439] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1747302029850435, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-05-15 09:40:29.852770] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.854163] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:29.862881] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.863244] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=28][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:11, tid:4602}]) [2025-05-15 09:40:29.872995] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.883108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.887141] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.887167] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14384559, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.887185] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14384559, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.887203] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.887220] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.887237] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.887247] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.887266] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.887285] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029887284, replica_locations:[]}) [2025-05-15 09:40:29.887307] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.887330] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.888542] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.888566] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.888586] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.888612] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.888629] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.888638] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.888656] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.888678] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029888677, replica_locations:[]}) [2025-05-15 09:40:29.888744] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=14382938, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=203, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.893218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.900051] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:29.900841] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.900866] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029900834) [2025-05-15 09:40:29.900880] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302029700793, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.900906] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:29.900942] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12332149373, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.900974] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.900998] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.901016] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.901025] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.901066] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35] get wrs ts(ls_id={id:1001}, delta=12332149373, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:29.901082] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.901099] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:29.901109] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:29.901121] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:29.901132] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.901146] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:29.901154] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029900926) [2025-05-15 09:40:29.902869] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:29.902868] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=37] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:29.903331] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.903392] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:29.912069] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.912858] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.912880] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.912899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.912920] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.912936] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.912956] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.912974] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.912985] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302029912835, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.913002] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.913010] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:29.913078] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:29.913092] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:29.913101] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:29.913116] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:29.913447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.922618] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12332193873, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:29.922645] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.922758] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=112][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:29.922778] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.922787] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.922805] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.922818] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:29.922827] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029922600) [2025-05-15 09:40:29.922866] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:29.922893] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302029922861) [2025-05-15 09:40:29.922912] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302029722810, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:29.922951] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12332022439, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:29.922970] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:29.922992] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:29.923003] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:29.923018] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:29.923029] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.923043] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:29.923050] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302029922929) [2025-05-15 09:40:29.923566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.932032] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4808951011, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:29.932066] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:29.932077] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1ab65efee0) [2025-05-15 09:40:29.932093] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4808951011, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:29.933676] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.943788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.953905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.955554] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:29.957978] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:29.958007] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:29.958025] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:29.958045] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:29.958066] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11792848397, flush_io_finished_ret_:0, flush_io_finished_round_:1504, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:29.963465] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:34, tid:3961}]) [2025-05-15 09:40:29.963607] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=41] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:29.963656] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:29.964020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.974116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.984226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.988870] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.988894] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14282829, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.988908] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14282829, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.988923] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.988933] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14282829, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.988951] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.988972] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.988992] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.989010] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.989020] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.989039] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.989058] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029989057, replica_locations:[]}) [2025-05-15 09:40:29.989079] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:29.989106] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:29.989121] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:29.989148] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.989201] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11793879677, cache_obj->added_lc()=false, cache_obj->get_object_id()=15726, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:29.990303] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:29.990327] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.990345] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:29.990361] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:29.990370] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:29.990387] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:29.990405] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:29.990423] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.990440] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:29.990450] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:29.990468] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:29.990479] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302029990478, replica_locations:[]}) [2025-05-15 09:40:29.990550] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14281131, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=204, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:29.994336] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:29.995345] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD2-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.995367] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD2-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.995384] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD2-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.995404] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD2-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.995423] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD2-0-0] [lt=18][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.995437] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:29.995454] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.995463] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.995478] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302029995321, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:29.995502] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.995511] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:29.995587] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:29.996347] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:29.996367] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:29.996376] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:29.996390] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:29.996398] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:29.996413] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:29.996431] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:29.996443] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:29.996501] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:29.996512] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:29.996816] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD3-0-0] [lt=15][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.996837] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD3-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:29.996848] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD3-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:29.996864] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD3-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:29.996872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD3-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:29.996883] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:29.996896] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:29.996905] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:29.996924] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302029996807, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:29.996934] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:29.996946] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:29.996983] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:29.998260] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=22] gc stale ls task succ [2025-05-15 09:40:30.000104] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.000916] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12332248726, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.000942] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.000970] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.000991] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.001007] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.001022] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12332249783, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.001039] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.001056] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.001066] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.001079] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.001089] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.001104] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.001126] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030000897) [2025-05-15 09:40:30.004450] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.005361] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15] ====== tenant freeze timer task ====== [2025-05-15 09:40:30.012918] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.012941] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.012957] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.012976] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.012992] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.013010] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.013026] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.013035] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.013049] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302030012897, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.013065] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.013072] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.013135] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:30.013148] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:30.013156] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:30.013169] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:30.014573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.022670] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.022703] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030022665) [2025-05-15 09:40:30.022754] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=51][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302029822642, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.022790] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12332294279, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.022811] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.022834] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.022853] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.022870] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.022882] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.022895] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.022902] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030022776) [2025-05-15 09:40:30.022929] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12332121786, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.022952] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.022974] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:30.022992] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.023000] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.023018] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.023032] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.023040] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030022918) [2025-05-15 09:40:30.024685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.027231] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:30.034797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.044910] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.048638] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [4329][T1_ObXAHbWorker][T1][Y0-0000000000000000-0-0] [lt=22] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-15 09:40:30.050426] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.050468] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:30.050478] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC966C-0-0] [lt=21][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302030050451}) [2025-05-15 09:40:30.050506] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:30.050527] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:30.050641] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:30.051024] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.051038] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:30.051053] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:30.055021] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.056350] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552741, support_ssl=false) [2025-05-15 09:40:30.056431] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] sm conn init succ(conn.sessid_=3221552741, sess.client_addr_="172.22.0.1:44904") [2025-05-15 09:40:30.056447] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552741, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.056643] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=34] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.056681] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.056710] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.056732] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.056746] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.056757] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.056780] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.056794] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552741, proxy_sessid=0) [2025-05-15 09:40:30.056822] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=10][errcode=0] server close connection(sessid=3221552741, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.056833] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552741, proxy_sessid=0) [2025-05-15 09:40:30.056842] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:30.056852] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9040-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552741, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.056897] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552741, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030055961, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.056917] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552741, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.056927] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552741, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030055961, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.056942] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552741) [2025-05-15 09:40:30.056961] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552741, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.065131] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.065521] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-05-15 09:40:30.065545] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:30.068042] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:30.068089] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.068126] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:30.068136] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:30.068175] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=39][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:30.068257] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:30.075246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.085367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.090678] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.090706] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14181022, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.090725] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14181022, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.090742] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.090766] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14181022, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.090784] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.090804] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.090823] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.090841] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.090856] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.090875] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.090894] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030090892, replica_locations:[]}) [2025-05-15 09:40:30.090916] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.090943] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.090958] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:30.090980] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.091028] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11793981508, cache_obj->added_lc()=false, cache_obj->get_object_id()=15727, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.092237] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.092262] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.092279] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.092297] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.092304] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.092321] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.092340] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.092356] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.092372] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.092381] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.092399] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.092415] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030092414, replica_locations:[]}) [2025-05-15 09:40:30.092483] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14179199, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=205, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.093012] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221552742, support_ssl=false) [2025-05-15 09:40:30.093097] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552743, support_ssl=false) [2025-05-15 09:40:30.093092] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=47] sm conn init succ(conn.sessid_=3221552742, sess.client_addr_="172.22.0.1:44908") [2025-05-15 09:40:30.093138] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552743, sess.client_addr_="172.22.0.1:44916") [2025-05-15 09:40:30.093121] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552742, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093156] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552743, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093319] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.093325] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=37] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.093349] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.093382] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=26] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.093385] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.093383] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552744, support_ssl=false) [2025-05-15 09:40:30.093408] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.093404] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.093418] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.093412] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221552745, support_ssl=false) [2025-05-15 09:40:30.093423] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.093427] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.093434] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.093448] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.093451] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552745, sess.client_addr_="172.22.0.1:44938") [2025-05-15 09:40:30.093452] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093467] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552745, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093473] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093475] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552743, proxy_sessid=0) [2025-05-15 09:40:30.093499] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=24][errcode=-4018] get session fail(ret=-4018, sessid=3221552742, proxy_sessid=0) [2025-05-15 09:40:30.093519] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552744, sess.client_addr_="172.22.0.1:44924") [2025-05-15 09:40:30.093558] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=22][errcode=0] server close connection(sessid=3221552743, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093559] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=10][errcode=0] server close connection(sessid=3221552742, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093547] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552744, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093574] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552743, proxy_sessid=0) [2025-05-15 09:40:30.093573] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552742, proxy_sessid=0) [2025-05-15 09:40:30.093582] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:30.093581] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.093592] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907B-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552742, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.093592] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904A-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552743, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.093641] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.093671] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.093655] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552743, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093693] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=34] kill and revert session(conn.sessid_=3221552743, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.093696] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.093696] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.093702] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552742, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093715] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.093713] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552743, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.093727] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.093733] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552743) [2025-05-15 09:40:30.093729] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552742, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.093734] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.093745] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.093744] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552742, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.093753] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.093756] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093748] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552743, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.093763] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552742) [2025-05-15 09:40:30.093766] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.093769] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552745, proxy_sessid=0) [2025-05-15 09:40:30.093775] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.093783] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.093792] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=9][errcode=0] server close connection(sessid=3221552745, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093783] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552742, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.093803] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552745, proxy_sessid=0) [2025-05-15 09:40:30.093802] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093813] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:30.093816] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552744, proxy_sessid=0) [2025-05-15 09:40:30.093821] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9041-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552745, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.093834] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=8][errcode=0] server close connection(sessid=3221552744, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.093846] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552744, proxy_sessid=0) [2025-05-15 09:40:30.093856] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:30.093870] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904B-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552744, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.093880] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552745, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093902] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552745, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.093924] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552744, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.093943] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552745, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.093952] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552744, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.093961] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552745) [2025-05-15 09:40:30.093965] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552744, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302030092954, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.093977] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552745, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.093990] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552744) [2025-05-15 09:40:30.094017] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552744, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.095203] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552746, support_ssl=false) [2025-05-15 09:40:30.095210] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552747, support_ssl=false) [2025-05-15 09:40:30.095219] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552748, support_ssl=false) [2025-05-15 09:40:30.095247] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552747, sess.client_addr_="172.22.0.1:44962") [2025-05-15 09:40:30.095260] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552746, sess.client_addr_="172.22.0.1:44956") [2025-05-15 09:40:30.095264] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552747, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.095273] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221552748, sess.client_addr_="172.22.0.1:44946") [2025-05-15 09:40:30.095276] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552746, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.095298] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552748, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.095436] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.095472] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.095474] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.095482] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.095497] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.095505] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.095513] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.095516] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.095524] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.095534] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.095541] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.095544] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.095555] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.095558] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.095569] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.095571] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552748, proxy_sessid=0) [2025-05-15 09:40:30.095572] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.095580] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.095585] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.095588] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.095594] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=11][errcode=0] server close connection(sessid=3221552748, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.095593] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.095608] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552748, proxy_sessid=0) [2025-05-15 09:40:30.095605] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.095615] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.095612] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.095620] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552746, proxy_sessid=0) [2025-05-15 09:40:30.095625] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552747, proxy_sessid=0) [2025-05-15 09:40:30.095622] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907C-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552748, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.095643] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=9][errcode=0] server close connection(sessid=3221552746, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.095653] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552746, proxy_sessid=0) [2025-05-15 09:40:30.095652] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=17][errcode=0] server close connection(sessid=3221552747, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.095664] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:30.095665] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552747, proxy_sessid=0) [2025-05-15 09:40:30.095673] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.095671] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9042-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552746, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.095671] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552748, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302030095068, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.095692] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552748, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.095686] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904C-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552747, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.095702] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552748, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302030095068, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.095719] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552748) [2025-05-15 09:40:30.095725] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552746, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030095068, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.095733] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552747, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030095068, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.095736] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552748, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.095751] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552746, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.095757] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552747, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.095759] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552746, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030095068, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.095768] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552747, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030095068, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.095786] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552747) [2025-05-15 09:40:30.095784] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] mark session id unused(conn.sessid_=3221552746) [2025-05-15 09:40:30.095800] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552747, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.095804] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552746, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.097715] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552749, support_ssl=false) [2025-05-15 09:40:30.097762] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552749, sess.client_addr_="172.22.0.1:44976") [2025-05-15 09:40:30.097787] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552749, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.097919] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.097952] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.097969] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.097988] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.097996] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.098010] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.098023] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.098039] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552749, proxy_sessid=0) [2025-05-15 09:40:30.098060] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=10][errcode=0] server close connection(sessid=3221552749, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.098073] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552749, proxy_sessid=0) [2025-05-15 09:40:30.098083] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:30.098096] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907D-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552749, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.098146] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552749, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302030097184, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.098181] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=33] kill and revert session(conn.sessid_=3221552749, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.098193] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552749, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302030097184, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.098212] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552749) [2025-05-15 09:40:30.098225] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552749, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.099029] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552750, support_ssl=false) [2025-05-15 09:40:30.099040] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552751, support_ssl=false) [2025-05-15 09:40:30.099083] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552750, sess.client_addr_="172.22.0.1:44980") [2025-05-15 09:40:30.099094] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552751, sess.client_addr_="172.22.0.1:44992") [2025-05-15 09:40:30.099101] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552750, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.099109] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552751, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.099136] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552752, support_ssl=false) [2025-05-15 09:40:30.099180] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221552752, sess.client_addr_="172.22.0.1:45004") [2025-05-15 09:40:30.099210] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552752, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.099251] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.099285] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.099291] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.099303] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.099321] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.099323] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.099340] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.099332] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.099347] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.099348] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.099364] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.099363] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.099371] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.099375] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552751, proxy_sessid=0) [2025-05-15 09:40:30.099389] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.099397] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=9][errcode=0] server close connection(sessid=3221552751, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.099405] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552750, proxy_sessid=0) [2025-05-15 09:40:30.099408] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552751, proxy_sessid=0) [2025-05-15 09:40:30.099415] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:30.099425] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=10][errcode=0] server close connection(sessid=3221552750, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.099422] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9043-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552751, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.099431] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.099442] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552750, proxy_sessid=0) [2025-05-15 09:40:30.099449] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.099452] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.099454] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.099464] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.099457] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904D-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552750, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.099462] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552751, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030098242, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.099476] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.099482] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552751, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.099482] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.099501] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552751, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030098242, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.099508] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=22] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.099518] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552751) [2025-05-15 09:40:30.099517] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552752, proxy_sessid=0) [2025-05-15 09:40:30.099510] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552750, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030098242, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.099535] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552750, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.099536] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=10][errcode=0] server close connection(sessid=3221552752, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.099539] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552751, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.099550] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552752, proxy_sessid=0) [2025-05-15 09:40:30.099561] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:30.099550] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552750, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030098242, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.099573] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221552750) [2025-05-15 09:40:30.099571] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9044-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552752, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.099594] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221552750, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.099636] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552752, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302030099302, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.099668] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221552752, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.099718] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552752, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302030099302, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.099741] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221552752) [2025-05-15 09:40:30.099754] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552752, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.100169] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.100962] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.100990] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030100956) [2025-05-15 09:40:30.101008] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302029900897, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.101007] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13] ====== tenant freeze timer task ====== [2025-05-15 09:40:30.101043] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12332349165, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.101064] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.101090] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.101111] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.101126] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.101139] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12332349165, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.101156] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.101173] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.101183] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.101195] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.101206] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.101220] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.101228] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030101025) [2025-05-15 09:40:30.103659] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:30.105615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.113064] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.113087] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.113108] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.113129] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.113144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.113161] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.113171] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.113181] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.113195] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302030113037, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.113210] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.113219] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.113296] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:30.113310] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:30.113319] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:30.113333] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:30.114476] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:30.114505] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:30.114516] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=9] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:30.114536] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=20] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:30.115725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.119818] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=16] gc stale ls task succ [2025-05-15 09:40:30.122705] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12332394747, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.122733] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.122813] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=78][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.122835] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.122845] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.122866] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.122881] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.122888] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030122681) [2025-05-15 09:40:30.122984] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.123026] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030122978) [2025-05-15 09:40:30.123045] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302029922928, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.123074] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12332222254, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.123094] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.123115] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:30.123127] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.123141] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.123156] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.123169] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.123176] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030123061) [2025-05-15 09:40:30.125839] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.127215] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:30.127242] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:30.127254] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:30.127275] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:30.127295] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11793017634, flush_io_finished_ret_:0, flush_io_finished_round_:1504, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:30.135954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.146066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.156185] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.166299] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.173337] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.173706] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.174434] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=27] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.174589] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:30.174627] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.174628] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=37][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:30.174647] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:30.174669] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=20][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:30.174688] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:30.174701] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:30.174714] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:30.174729] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:30.174739] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:30.174754] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=15][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:30.174763] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:30.174777] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:30.174787] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:30.174801] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:30.174829] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=17][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:30.174845] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.174862] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.174877] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:30.174892] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=13][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:30.174909] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:30.174923] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:30.174948] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=21][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:30.174972] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:30.174986] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:30.174994] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:30.175020] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:30.175037] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6451-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.175046] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:30.175063] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:30.175071] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:30.175085] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=13][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:30.175093] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302030174287, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:30.175109] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:30.175119] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:30.175202] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:30.175220] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=16][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:30.175231] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=10][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:30.175246] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:30.175258] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:30.175275] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=14][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:30.175283] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6451-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:30.176418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.180384] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=18] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-15 09:40:30.180414] INFO [COMMON] handle (memory_dump.cpp:531) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-15 09:40:30.180781] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] update_check_range(min_check_version=149, max_check_version=150, global_version=151) [2025-05-15 09:40:30.186532] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.190922] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=20] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:30.192632] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.192658] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14079069, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.192678] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=14079069, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.192695] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.192705] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=14079069, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.192723] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.192737] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.192756] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.192774] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.192784] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.192801] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.192820] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030192818, replica_locations:[]}) [2025-05-15 09:40:30.192842] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.192870] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.192884] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:30.192924] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.192972] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11794083450, cache_obj->added_lc()=false, cache_obj->get_object_id()=15728, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.194089] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.194112] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.194126] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.194141] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.194150] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.194168] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.194187] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.194204] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.194221] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.194236] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.194249] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.194265] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030194265, replica_locations:[]}) [2025-05-15 09:40:30.194332] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=14077349, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=206, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.195240] INFO handle (memory_dump.cpp:658) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=16] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 116 220 0 1 2 57 1 0 1 4 1 1 0 1 5 1 3 0 1 7 1 2 0 1 8 43 1 0 1 12 3 2 0 1 13 2 1 0 1 18 11 3 0 1 20 1 1 0 500 0 57 222 0 500 7 9 4 0 500 8 49 1 0 500 9 2 1 0 500 10 2 1 0 500 17 6 3 0 500 22 3 50 0 500 23 23 10 0 508 0 2 7 0 508 8 15 1 0 509 0 2 7 0 509 8 9 1 0 1001 0 76 205 0 1001 2 35 1 0 1001 4 1 1 0 1001 5 1 1 0 1001 8 29 1 0 1001 12 3 3 0 1001 13 1 1 0 1001 18 17 3 0 1001 20 2 1 0 1002 0 82 206 0 1002 2 90 1 0 1002 4 1 1 0 1002 5 1 1 0 1002 8 63 1 0 1002 12 2 2 0 1002 13 1 1 0 1002 18 5 3 0 cost_time: 14511 [2025-05-15 09:40:30.195288] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=27] [MEMORY] tenant: 1, limit: 5368709120 hold: 850857984 cache_hold: 140509184 cache_used: 140509184 cache_item_count: 67 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 410435584 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 119537664 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 90177536 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 6291456 limit= 1073741820 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 4194304 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 37769216 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [2025-05-15 09:40:30.195579] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=17] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 410435584 used= 397587936 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 82821120 used= 82569792 count= 62 avg_used= 1331770 block_cnt= 62 chunk_cnt= 44 mod=MysqlRequesReco [MEMORY] hold= 41963520 used= 41943040 count= 1 avg_used= 41943040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31477760 used= 31458304 count= 1 avg_used= 31458304 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 31428608 used= 31368256 count= 7 avg_used= 4481179 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 28008448 used= 27926528 count= 4 avg_used= 6981632 block_cnt= 4 chunk_cnt= 4 mod=ServerObjecPool [MEMORY] hold= 16232912 used= 16067217 count= 111 avg_used= 144749 block_cnt= 38 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 9859072 used= 9831040 count= 2 avg_used= 4915520 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 9510912 used= 9437472 count= 9 avg_used= 1048608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 8777728 used= 8760064 count= 2 avg_used= 4380032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 5386240 used= 5368320 count= 1 avg_used= 5368320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 5275648 used= 4456480 count= 161 avg_used= 27680 block_cnt= 161 chunk_cnt= 21 mod=DiagnosticInfo [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 4235264 used= 4194304 count= 2 avg_used= 2097152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4214784 used= 4194336 count= 1 avg_used= 4194336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 block_cnt= 2 chunk_cnt= 2 mod=CACHE_MAP_NODE [MEMORY] hold= 3879456 used= 3842280 count= 89 avg_used= 43171 block_cnt= 89 chunk_cnt= 4 mod=TabletMap [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMem [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3772416 used= 3751944 count= 1 avg_used= 3751944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3723264 used= 3698704 count= 2 avg_used= 1849352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3162112 used= 3145808 count= 2 avg_used= 1572904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3145728 used= 2228224 count= 128 avg_used= 17408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 3047424 used= 3013468 count= 10 avg_used= 301346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3002368 used= 2982720 count= 1 avg_used= 2982720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2375680 used= 2359536 count= 2 avg_used= 1179768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2248704 used= 2228224 count= 1 avg_used= 2228224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2198768 used= 416700 count= 25277 avg_used= 16 block_cnt= 271 chunk_cnt= 3 mod=Number [MEMORY] hold= 2150400 used= 2131072 count= 1 avg_used= 2131072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 2097152 used= 2087424 count= 2 avg_used= 1043712 block_cnt= 2 chunk_cnt= 2 mod=DiagnosticSum [MEMORY] hold= 1843200 used= 1770696 count= 9 avg_used= 196744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1584544 used= 1575992 count= 3 avg_used= 525330 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1269568 used= 1263904 count= 6 avg_used= 210650 block_cnt= 6 chunk_cnt= 3 mod=PoolFreeList [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 958464 used= 950272 count= 1 avg_used= 950272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 786688 used= 524352 count= 33 avg_used= 15889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 739648 used= 722176 count= 91 avg_used= 7936 block_cnt= 91 chunk_cnt= 8 mod=TltMemtablMgr [MEMORY] hold= 737280 used= 655360 count= 10 avg_used= 65536 block_cnt= 10 chunk_cnt= 6 mod=[T]char [MEMORY] hold= 663552 used= 655360 count= 1 avg_used= 655360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663552 used= 659200 count= 1 avg_used= 659200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 598016 used= 590232 count= 1 avg_used= 590232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 491520 used= 468576 count= 3 avg_used= 156192 block_cnt= 3 chunk_cnt= 2 mod=SQLSessionInfo [MEMORY] hold= 475136 used= 458816 count= 2 avg_used= 229408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 417664 used= 412408 count= 4 avg_used= 103102 block_cnt= 4 chunk_cnt= 1 mod=ResultSet [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401408 used= 393504 count= 1 avg_used= 393504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 401408 used= 393248 count= 1 avg_used= 393248 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 401408 used= 393248 count= 1 avg_used= 393248 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 385024 used= 375520 count= 2 avg_used= 187760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 286720 used= 280000 count= 1 avg_used= 280000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 246880 used= 3318 count= 3061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212992 used= 211072 count= 1 avg_used= 211072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204800 used= 196744 count= 1 avg_used= 196744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 196608 used= 195840 count= 3 avg_used= 65280 block_cnt= 3 chunk_cnt= 1 mod=MemTblObj [MEMORY] hold= 180224 used= 105600 count= 11 avg_used= 9600 block_cnt= 11 chunk_cnt= 6 mod=[T]ObDSActionAr [MEMORY] hold= 147456 used= 139264 count= 1 avg_used= 139264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147328 used= 141880 count= 3 avg_used= 47293 block_cnt= 3 chunk_cnt= 1 mod=CommSysVarFac [MEMORY] hold= 122880 used= 110608 count= 2 avg_used= 55304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=DbmsSched_Job [MEMORY] hold= 114688 used= 106496 count= 1 avg_used= 106496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 104000 used= 101504 count= 13 avg_used= 7808 block_cnt= 13 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 98304 used= 94752 count= 1 avg_used= 94752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 97472 used= 84160 count= 3 avg_used= 28053 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 82080 used= 22016 count= 299 avg_used= 73 block_cnt= 63 chunk_cnt= 18 mod=Coro [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73728 used= 66176 count= 1 avg_used= 66176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65408 used= 63488 count= 4 avg_used= 15872 block_cnt= 4 chunk_cnt= 2 mod=IOUsageInfo [MEMORY] hold= 57344 used= 51328 count= 1 avg_used= 51328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 49152 used= 42496 count= 1 avg_used= 42496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48960 used= 44840 count= 2 avg_used= 22420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 47360 used= 45440 count= 10 avg_used= 4544 block_cnt= 10 chunk_cnt= 6 mod=[T]ObTraceEvent [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40640 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 37968 used= 4056 count= 169 avg_used= 24 block_cnt= 46 chunk_cnt= 16 mod=[T]MemoryContex [MEMORY] hold= 32704 used= 25191 count= 2 avg_used= 12595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 25600 used= 17920 count= 40 avg_used= 448 block_cnt= 16 chunk_cnt= 3 mod=OMT_Worker [MEMORY] hold= 24576 used= 16392 count= 1 avg_used= 16392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24384 used= 23808 count= 3 avg_used= 7936 block_cnt= 3 chunk_cnt= 1 mod=SqlSession [MEMORY] hold= 17024 used= 8448 count= 3 avg_used= 2816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=PlGdbHelper [MEMORY] hold= 16384 used= 14016 count= 1 avg_used= 14016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16384 used= 11584 count= 1 avg_used= 11584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16384 used= 9792 count= 1 avg_used= 9792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16384 used= 13896 count= 1 avg_used= 13896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16384 used= 10248 count= 1 avg_used= 10248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16384 used= 8200 count= 1 avg_used= 8200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16384 used= 8448 count= 1 avg_used= 8448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TLD_AssignedMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 9456 used= 9064 count= 2 avg_used= 4532 block_cnt= 2 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=RawPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CtxBlockPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ConstPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DictPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsTableMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=PlanVaIdx [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiVarMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8000 used= 7808 count= 1 avg_used= 7808 block_cnt= 1 chunk_cnt= 1 mod=HashNodeLCSta [MEMORY] hold= 7296 used= 6912 count= 2 avg_used= 3456 block_cnt= 2 chunk_cnt= 2 mod=CACHE_INST [MEMORY] hold= 5056 used= 4656 count= 2 avg_used= 2328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4848 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4288 used= 4096 count= 1 avg_used= 4096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4160 used= 3520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3328 used= 1792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3200 used= 3008 count= 1 avg_used= 3008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2880 used= 2400 count= 3 avg_used= 800 block_cnt= 3 chunk_cnt= 2 mod=DICollector [MEMORY] hold= 2592 used= 2016 count= 3 avg_used= 672 block_cnt= 2 chunk_cnt= 2 mod=[T]ObWarningBuf [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2048 used= 1856 count= 1 avg_used= 1856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2016 used= 1824 count= 1 avg_used= 1824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1920 used= 1728 count= 1 avg_used= 1728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1808 used= 1424 count= 2 avg_used= 712 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1280 used= 1088 count= 1 avg_used= 1088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1248 used= 1056 count= 1 avg_used= 1056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1232 used= 1032 count= 1 avg_used= 1032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1200 used= 1000 count= 1 avg_used= 1000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=PackStateMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceIdMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=ContextsMap [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 336 used= 144 count= 1 avg_used= 144 block_cnt= 1 chunk_cnt= 1 mod=ObTxLSLogCb [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 397587936 used= 391158529 count= 31478 avg_used= 12426 mod=SUMMARY [2025-05-15 09:40:30.195612] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=24] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 119537664 used= 118603776 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 118603776 used= 118549512 count= 57 avg_used= 2079816 block_cnt= 57 chunk_cnt= 57 mod=Memstore [MEMORY] hold= 118603776 used= 118549512 count= 57 avg_used= 2079816 mod=SUMMARY [2025-05-15 09:40:30.195628] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2097152 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460416 count= 1 avg_used= 460416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466944 used= 460416 count= 1 avg_used= 460416 mod=SUMMARY [2025-05-15 09:40:30.195643] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 2097152 used= 36928 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32768 used= 25864 count= 1 avg_used= 25864 block_cnt= 1 chunk_cnt= 1 mod=SqlPhyPlan [MEMORY] hold= 2176 used= 1984 count= 1 avg_used= 1984 block_cnt= 1 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 1984 used= 1600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 36928 used= 29448 count= 4 avg_used= 7362 mod=SUMMARY [2025-05-15 09:40:30.195666] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2097152 used= 80496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80400 used= 58751 count= 259 avg_used= 226 block_cnt= 20 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80496 used= 58783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-15 09:40:30.195686] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 90177536 used= 88768512 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 88768512 used= 88603392 count= 172 avg_used= 515136 block_cnt= 172 chunk_cnt= 43 mod=CoStack [MEMORY] hold= 88768512 used= 88603392 count= 172 avg_used= 515136 mod=SUMMARY [2025-05-15 09:40:30.195710] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 140509184 used= 140509184 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 140509184 used= 0 count= 67 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 140509184 used= 0 count= 67 avg_used= 0 mod=SUMMARY [2025-05-15 09:40:30.195731] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 6291456 used= 5775360 limit= 1073741820 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5373952 used= 5352960 count= 82 avg_used= 65280 block_cnt= 82 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 401408 used= 400064 count= 2 avg_used= 200032 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 5775360 used= 5753024 count= 84 avg_used= 68488 mod=SUMMARY [2025-05-15 09:40:30.195750] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 4194304 used= 2869184 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2869184 used= 2801408 count= 353 avg_used= 7936 block_cnt= 353 chunk_cnt= 2 mod=MemtableCallbac [MEMORY] hold= 2869184 used= 2801408 count= 353 avg_used= 7936 mod=SUMMARY [2025-05-15 09:40:30.195782] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 37769216 used= 35215872 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17862592 used= 17833792 count= 3 avg_used= 5944597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 17345152 used= 16935424 count= 2134 avg_used= 7936 block_cnt= 2134 chunk_cnt= 10 mod=TX_DATA_SLICE [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 35215872 used= 34777152 count= 2138 avg_used= 16266 mod=SUMMARY [2025-05-15 09:40:30.195806] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2097152 used= 186944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 186944 used= 182528 count= 23 avg_used= 7936 block_cnt= 23 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 186944 used= 182528 count= 23 avg_used= 7936 mod=SUMMARY [2025-05-15 09:40:30.195831] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 33554432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-05-15 09:40:30.195856] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 580640768 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 186875904 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 18874368 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 102760448 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4194304 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 6291456 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 17969152 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11292672 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 232382464 limit= 9223372036854775807 [2025-05-15 09:40:30.196122] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 186875904 used= 172036096 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31428608 used= 31368256 count= 7 avg_used= 4481179 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 12873728 used= 12856400 count= 1 avg_used= 12856400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckInteChe [MEMORY] hold= 8933376 used= 8912896 count= 1 avg_used= 8912896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7275200 used= 6334546 count= 1107 avg_used= 5722 block_cnt= 574 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5562368 used= 5551360 count= 40 avg_used= 138784 block_cnt= 37 chunk_cnt= 16 mod=TenantCtxAlloca [MEMORY] hold= 5533696 used= 5456720 count= 8 avg_used= 682090 block_cnt= 8 chunk_cnt= 4 mod=KvstCachWashStr [MEMORY] hold= 5283840 used= 5243040 count= 5 avg_used= 1048608 block_cnt= 5 chunk_cnt= 5 mod=ConcurHashMap [MEMORY] hold= 5242912 used= 5150064 count= 47 avg_used= 109575 block_cnt= 44 chunk_cnt= 12 mod=PartitTableTask [MEMORY] hold= 4747072 used= 4726520 count= 2 avg_used= 2363260 block_cnt= 2 chunk_cnt= 2 mod=HashBucTenComMo [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=PxP2PDhMgrKey [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=HashPxBlooFilKe [MEMORY] hold= 4307456 used= 4278711 count= 3 avg_used= 1426237 block_cnt= 3 chunk_cnt= 3 mod=SqlDtlMgr [MEMORY] hold= 4232240 used= 4203008 count= 6 avg_used= 700501 block_cnt= 6 chunk_cnt= 5 mod=BaseLogWriter [MEMORY] hold= 4214896 used= 4194328 count= 2 avg_used= 2097164 block_cnt= 2 chunk_cnt= 2 mod=SerFuncRegHT [MEMORY] hold= 4194304 used= 4176267 count= 1 avg_used= 4176267 block_cnt= 1 chunk_cnt= 1 mod=SyslogCompress [MEMORY] hold= 3981312 used= 3903840 count= 12 avg_used= 325320 block_cnt= 12 chunk_cnt= 5 mod=DedupQueue [MEMORY] hold= 3829760 used= 3801088 count= 2 avg_used= 1900544 block_cnt= 2 chunk_cnt= 2 mod=TabletLSMap [MEMORY] hold= 2532032 used= 2513495 count= 3 avg_used= 837831 block_cnt= 3 chunk_cnt= 3 mod=FtParse [MEMORY] hold= 2383776 used= 2342304 count= 6 avg_used= 390384 block_cnt= 6 chunk_cnt= 5 mod=Omt [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdUnitMa [MEMORY] hold= 2150400 used= 2113528 count= 3 avg_used= 704509 block_cnt= 3 chunk_cnt= 2 mod=FixedQueue [MEMORY] hold= 1835520 used= 1674496 count= 844 avg_used= 1984 block_cnt= 285 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1581056 used= 1572904 count= 1 avg_used= 1572904 block_cnt= 1 chunk_cnt= 1 mod=IdConnMap [MEMORY] hold= 1522112 used= 1498368 count= 45 avg_used= 33297 block_cnt= 45 chunk_cnt= 16 mod=CommonArray [MEMORY] hold= 1286144 used= 1270296 count= 2 avg_used= 635148 block_cnt= 2 chunk_cnt= 2 mod=PxResMgr [MEMORY] hold= 1277952 used= 1260096 count= 3 avg_used= 420032 block_cnt= 3 chunk_cnt= 3 mod=CompEventMgr [MEMORY] hold= 1179648 used= 1048576 count= 16 avg_used= 65536 block_cnt= 16 chunk_cnt= 8 mod=[T]char [MEMORY] hold= 1073984 used= 1039504 count= 134 avg_used= 7757 block_cnt= 134 chunk_cnt= 3 mod=MallocInfoMap [MEMORY] hold= 1056768 used= 1048608 count= 1 avg_used= 1048608 block_cnt= 1 chunk_cnt= 1 mod=TsSourceInfoMap [MEMORY] hold= 1056768 used= 1011712 count= 6 avg_used= 168618 block_cnt= 6 chunk_cnt= 5 mod=LightyQueue [MEMORY] hold= 1056768 used= 1048608 count= 1 avg_used= 1048608 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 941872 used= 925368 count= 3 avg_used= 308456 block_cnt= 3 chunk_cnt= 3 mod=CACHE_INST_MAP [MEMORY] hold= 938576 used= 880440 count= 13 avg_used= 67726 block_cnt= 13 chunk_cnt= 8 mod=HashBucket [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSLocationMap [MEMORY] hold= 663552 used= 643296 count= 3 avg_used= 214432 block_cnt= 3 chunk_cnt= 3 mod=CompSuggestMgr [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=GEleTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=EventTimer [MEMORY] hold= 633856 used= 619008 count= 78 avg_used= 7936 block_cnt= 78 chunk_cnt= 5 mod=ModulePageAlloc [MEMORY] hold= 622592 used= 619008 count= 1 avg_used= 619008 block_cnt= 1 chunk_cnt= 1 mod=SysTaskStatus [MEMORY] hold= 592320 used= 579328 count= 73 avg_used= 7936 block_cnt= 73 chunk_cnt= 14 mod=LinkHashNodeget_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:30.252926] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=22] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:30.257684] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:30.257721] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=30] temporary file module use no blocks [2025-05-15 09:40:30.257736] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:30.257756] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:30.257776] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11793148093, flush_io_finished_ret_:0, flush_io_finished_round_:1504, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:30.258228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.264081] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=34][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:1, tid:4463}]) [2025-05-15 09:40:30.265750] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:30.265780] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:30.265800] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:30.265818] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:30.265834] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:30.265843] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:30.265861] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=13][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:30.265875] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=12][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:30.265884] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:30.265898] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=12][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:30.265906] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:30.265922] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=15][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:30.265931] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:30.265944] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=13][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:30.265965] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:30.265981] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.265997] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.266015] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=16][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:30.266032] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:30.266050] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=16][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:30.266059] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:30.266081] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:30.266105] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:30.266118] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:30.266125] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:30.266173] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=18][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:30.266197] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4420][T1001_Occam][T1][YB427F000001-000635291DAC623F-0-0] [lt=23][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.266208] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4420][T1001_Occam][T1001][YB427F000001-000635291DAC623F-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:30.266231] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:30.266250] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:30.266258] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:30.266273] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302030265504, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:30.266289] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:30.266300] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1001, tenant_id=1, columns=0x7f1aa10563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:30.266321] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1001, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:30.266408] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1aa10563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:30.266428] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:30.266437] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:30.266451] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:30.266474] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=15] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:30.268343] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.273258] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6453-0-0] [lt=16][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.273279] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6453-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.273296] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6453-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.273312] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6453-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.273321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6453-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.273335] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:30.273343] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.273356] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.273364] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302030273246, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:30.273379] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.273387] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:30.273435] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1001, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:30.273484] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:30.274674] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6454-0-0] [lt=17][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.274692] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6454-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.274705] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6454-0-0] [lt=7][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.274721] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6454-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.274728] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4460][T1001_TimerWK0_FreInfoReload][T1001][YB427F000001-000635291CFC6454-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.274742] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:30.274751] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.274759] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.274771] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302030274666, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:30.274780] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.274793] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:30.274829] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to select for update snapshot_gc_scn(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001) [2025-05-15 09:40:30.274844] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to load updated info(ret=-5627) [2025-05-15 09:40:30.274851] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to try update info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:30.278462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.288584] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.294467] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.294505] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=32][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13977233, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.294523] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13977233, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.294542] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.294552] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=13977233, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.294570] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.294586] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.294604] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.294623] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.294632] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.294651] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.294674] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030294671, replica_locations:[]}) [2025-05-15 09:40:30.294695] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.294725] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.294740] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:30.294763] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.294811] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11794185287, cache_obj->added_lc()=false, cache_obj->get_object_id()=15729, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.296059] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.296082] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.296095] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.296111] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.296119] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.296136] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.296155] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.296167] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.296183] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.296192] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.296210] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.296227] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030296226, replica_locations:[]}) [2025-05-15 09:40:30.296299] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13975383, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=207, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.296390] WDIAG load_file_to_string (utility.h:673) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:30.296413] WDIAG get_ethernet_speed (utility.cpp:575) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:30.296424] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:30.296433] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:30.298698] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.300288] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.301085] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.301125] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030301077) [2025-05-15 09:40:30.301145] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302030101024, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.301177] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12332548928, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.301200] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.301227] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.301246] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.301255] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.301275] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12332549986, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.301291] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.301309] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.301319] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.301332] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.301344] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.301358] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.301365] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030301157) [2025-05-15 09:40:30.303946] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:30.308817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.309188] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5212, clean_start_pos=12000000, clean_num=200000) [2025-05-15 09:40:30.313152] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.313177] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=25][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.313195] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.313215] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.313226] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.313269] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=42][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.313279] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.313313] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=34][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.313322] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302030313131, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.313338] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.313348] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.313413] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:30.313427] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:30.313435] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:30.313450] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:30.318929] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.322832] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12332594475, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.322863] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.322891] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.322913] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.322931] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.322944] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.322989] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=44][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.323005] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030322807) [2025-05-15 09:40:30.323105] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.323136] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030323097) [2025-05-15 09:40:30.323158] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302030123061, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.323192] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12332421982, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.323211] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.323233] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:30.323245] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.323254] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.323272] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.323286] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.323293] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030323178) [2025-05-15 09:40:30.329040] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.339142] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.339316] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.349254] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.351026] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4634][T1002_ArcSender][T1002][YB427F000001-000635291E2C6112-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:30.352420] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:30.352451] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.352468] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:30.352476] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:30.352503] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:30.352587] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=84][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:30.352598] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.352611] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:30.352618] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:30.352631] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:30.352690] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:30.355736] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:30.355809] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:30.355879] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:30.355945] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:30.355977] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:11744, total:11754) [2025-05-15 09:40:30.356018] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:30.356068] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=45] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:11744, total:11754) [2025-05-15 09:40:30.356100] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:30.356122] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:30.356141] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:30.356160] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628765][2nd=1892]}) [2025-05-15 09:40:30.359367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.364312] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=35][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:354, tid:4290}]) [2025-05-15 09:40:30.366210] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4579][T1002_TimerWK2_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=31] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):195, sequence:1}) [2025-05-15 09:40:30.369478] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.379600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.390756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.394874] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.395543] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.396217] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.396366] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=22] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.396425] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.396447] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13875274, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.396465] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13875274, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.396482] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.396477] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.396501] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=13875274, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.396544] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.396618] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=117][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.396633] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.396652] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.396669] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.396678] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.396696] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.396718] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030396716, replica_locations:[]}) [2025-05-15 09:40:30.396739] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.396760] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:207, local_retry_times:207, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:30.396785] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.396799] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:30.396798] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:30.396813] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.396826] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.396833] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:30.396871] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:30.396889] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.396932] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11794287410, cache_obj->added_lc()=false, cache_obj->get_object_id()=15730, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.398030] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.398061] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=30][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.398098] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.398116] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.398131] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.398147] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.398156] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.398172] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.398190] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.398208] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.398224] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.398233] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.398250] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.398267] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030398266, replica_locations:[]}) [2025-05-15 09:40:30.398288] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.398304] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.398314] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.398333] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:30.398341] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:30.398356] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:30.398376] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:30.398394] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.398402] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.398416] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:30.398426] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:30.398440] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:30.398450] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:30.398465] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:30.398473] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:30.398495] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:30.398504] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:30.398519] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:30.398527] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:30.398546] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:30.398560] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.398570] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:30.398584] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:30.398599] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:30.398607] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=208, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:30.398635] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] will sleep(sleep_us=100000, remain_us=13873046, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=208, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.400341] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.400900] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.401163] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12332649337, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.401187] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.401213] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.401232] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.401241] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.401261] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12332649337, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.401279] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.401297] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.401314] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.401321] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.401338] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.401352] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.401359] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030401147) [2025-05-15 09:40:30.411016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.412157] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:30.412182] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.412196] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:30.412225] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:30.413152] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.413175] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.413192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.413288] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=95][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.413304] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.413321] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.413338] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.413348] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.413364] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302030413132, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.413380] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.413405] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=25][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.413467] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:30.413480] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:30.413499] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:30.413515] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:30.418307] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.418330] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.418343] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.419401] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:30.419425] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.419437] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.419456] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:30.419470] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:30.419479] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:30.421129] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.422874] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.422901] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030422868) [2025-05-15 09:40:30.423076] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=178][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302030222845, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.423110] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12332694913, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.423135] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.423163] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.423167] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12332522420, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.423183] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.423191] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.423186] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.423207] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:30.423210] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.423220] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.423222] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.423235] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.423246] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.423260] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.423236] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030423094) [2025-05-15 09:40:30.423268] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030423153) [2025-05-15 09:40:30.427423] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-05-15 09:40:30.430166] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:30.430186] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.430198] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.431245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.441362] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.451472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.461588] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.470481] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:30.470532] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:30.471700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.481814] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.491931] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.496549] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.496589] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.496660] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=70][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:30.496670] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:30.496685] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:30.496697] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:30.496718] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:30.496764] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=44] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:30.496804] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:30.496810] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:30.496829] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.496840] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:30.496868] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:30.496883] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:30.496891] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:30.496906] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:30.496918] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:30.498761] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.498786] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13772937, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.498806] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13772937, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.498823] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.498832] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=13772937, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.498850] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.498870] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.498888] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.498907] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.498916] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.498931] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.498950] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030498948, replica_locations:[]}) [2025-05-15 09:40:30.498970] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.498991] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:208, local_retry_times:208, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:30.499014] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.499028] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:30.499042] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.499055] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.499061] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:30.499083] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:30.499099] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.499145] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11794389622, cache_obj->added_lc()=false, cache_obj->get_object_id()=15731, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.500269] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.500298] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.500334] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.500351] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.500368] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.500385] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.500394] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.500406] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6672-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.500410] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.500418] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.500423] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.500421] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6672-0-0] [lt=14][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.500453] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.500464] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6672-0-0] [lt=31][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.500469] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.500479] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6672-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.500479] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.500504] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6672-0-0] [lt=23][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.500504] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.500517] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:30.500523] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030500522, replica_locations:[]}) [2025-05-15 09:40:30.500533] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.500539] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.500544] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.500556] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.500557] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302030500371, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:30.500573] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.500573] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.500584] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:30.500587] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:30.500600] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:30.500610] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:30.500629] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:30.500646] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.500654] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.500668] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:30.500677] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:30.500692] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:30.500702] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:30.500673] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1002, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:30.500719] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:30.500721] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:30.500729] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:30.500737] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:30.500744] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:30.500752] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:30.500760] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:30.500779] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:30.500793] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.500812] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:30.500821] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:30.500836] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:30.500845] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=209, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:30.500872] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] will sleep(sleep_us=100000, remain_us=13770810, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=209, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.501203] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.501227] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030501197) [2025-05-15 09:40:30.501244] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302030301156, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.501279] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12332749730, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.501299] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.501324] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.501344] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.501354] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.501373] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12332749730, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.501390] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.501408] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.501417] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.501429] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.501439] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.501452] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.501460] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030501261) [2025-05-15 09:40:30.501949] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=8][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.501971] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.501980] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:30.501994] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:30.502001] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=7][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:30.502018] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=12][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:30.502033] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.502041] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6673-0-0] [lt=7][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5157, ret=-5157) [2025-05-15 09:40:30.502044] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.502055] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:30.502063] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:30.502071] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:30.502085] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78ca700, start=1747302030501939, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:30.502095] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:30.502107] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] execute sql failed(ret=-5157, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:30.502146] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to select for update snapshot_gc_scn(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.502161] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5157] failed to load updated info(ret=-5157) [2025-05-15 09:40:30.502169] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to try update info(tmp_ret=-5157, tmp_ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:30.509436] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:30.510940] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4291][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=35] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:30.512154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.513101] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.513122] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.513135] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.513152] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.513161] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.513176] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.513184] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.513198] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.513206] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302030513088, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.513225] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.513242] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.513289] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:30.513303] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:30.513313] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:30.513327] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:30.518371] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.518395] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.518409] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.519462] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:30.519503] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.519517] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.519539] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:30.519553] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:30.519563] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:30.522267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.522961] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37] get wrs ts(ls_id={id:1}, delta=12332794152, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.522992] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.523021] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.523179] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=157][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.523189] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.523208] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.523222] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.523218] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.523230] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030522933) [2025-05-15 09:40:30.523246] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030523211) [2025-05-15 09:40:30.523268] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302030323174, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.523297] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12332622707, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.523317] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.523341] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:30.523358] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.523366] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.523379] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.523392] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.523401] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030523285) [2025-05-15 09:40:30.530222] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:30.530244] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.530255] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.532379] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.537662] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=19] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:30.541583] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:30.542495] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.551503] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.551528] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.551555] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:30.551572] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:30.551588] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:30.551608] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] refresh gts failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:30.551623] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:30.551632] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:30.551646] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.551668] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:30.551688] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:30.551707] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:30.551717] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:30.551731] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:30.551739] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:30.551755] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:30.552107] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:30.552123] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.552138] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.552151] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:30.552159] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:30.552171] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:30.552177] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:30.552190] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.552204] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:30.554786] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.557535] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552765, support_ssl=false) [2025-05-15 09:40:30.557663] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221552765, sess.client_addr_="172.22.0.1:45010") [2025-05-15 09:40:30.557686] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552765, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.557873] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.557916] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.557943] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.557965] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.557981] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.557991] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.558021] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.558044] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552765, proxy_sessid=0) [2025-05-15 09:40:30.558088] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=15][errcode=0] server close connection(sessid=3221552765, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.558103] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552765, proxy_sessid=0) [2025-05-15 09:40:30.558110] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.558128] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907E-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552765, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.558185] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552765, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030557834, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.558215] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221552765, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.558225] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552765, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030557834, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.558246] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552765) [2025-05-15 09:40:30.558268] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552765, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.561566] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6690-0-0] [lt=24][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.561588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6690-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.561604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6690-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.561624] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6690-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.561635] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.561651] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:30.561661] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.561675] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.561684] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302030561542, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:30.561702] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.561711] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:30.561790] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=15][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:30.561807] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=15][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:30.561817] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=10][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:30.561831] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=14][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:30.561841] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=8][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:30.561856] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=13][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:30.561865] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6690-0-0] [lt=9][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:30.564906] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.575024] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.585140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.594581] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552767, support_ssl=false) [2025-05-15 09:40:30.594672] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=52] sm conn init succ(conn.sessid_=3221552767, sess.client_addr_="172.22.0.1:45020") [2025-05-15 09:40:30.594668] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552768, support_ssl=false) [2025-05-15 09:40:30.594703] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552767, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.594751] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=43] sm conn init succ(conn.sessid_=3221552768, sess.client_addr_="172.22.0.1:45028") [2025-05-15 09:40:30.594784] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552768, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.594869] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=43] new mysql sessid created(conn.sessid_=3221552769, support_ssl=false) [2025-05-15 09:40:30.594886] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] new mysql sessid created(conn.sessid_=3221552770, support_ssl=false) [2025-05-15 09:40:30.594923] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221552769, sess.client_addr_="172.22.0.1:45032") [2025-05-15 09:40:30.594939] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552769, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.594951] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552770, sess.client_addr_="172.22.0.1:45030") [2025-05-15 09:40:30.594973] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552770, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.594983] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=37] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.595002] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=46] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.595040] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.595056] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.595064] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.595087] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.595087] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.595097] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.595110] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.595111] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.595125] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.595133] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.595134] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.595136] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595153] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552767, proxy_sessid=0) [2025-05-15 09:40:30.595157] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595153] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.595165] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.595177] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=8][errcode=0] server close connection(sessid=3221552767, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595176] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552768, proxy_sessid=0) [2025-05-15 09:40:30.595177] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.595195] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.595193] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552767, proxy_sessid=0) [2025-05-15 09:40:30.595190] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.595203] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.595204] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:30.595211] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.595216] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=15][errcode=0] server close connection(sessid=3221552768, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595214] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904E-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552767, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.595224] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595230] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552768, proxy_sessid=0) [2025-05-15 09:40:30.595239] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:30.595237] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552770, proxy_sessid=0) [2025-05-15 09:40:30.595247] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.595258] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=9][errcode=0] server close connection(sessid=3221552770, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595261] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.595249] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9045-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552768, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.595259] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.595269] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.595268] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552770, proxy_sessid=0) [2025-05-15 09:40:30.595281] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.595281] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:30.595265] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552767, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.595300] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595298] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] kill and revert session(conn.sessid_=3221552767, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.595295] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC907F-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552770, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.595309] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552769, proxy_sessid=0) [2025-05-15 09:40:30.595310] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552767, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.595301] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552768, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.595330] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=12][errcode=0] server close connection(sessid=3221552769, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.595330] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552767) [2025-05-15 09:40:30.595336] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221552768, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.595345] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552769, proxy_sessid=0) [2025-05-15 09:40:30.595353] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.595343] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552770, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.595353] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552768, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.595360] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC904F-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552769, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.595373] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221552770, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.595355] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221552767, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.595378] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552768) [2025-05-15 09:40:30.595382] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552770, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.595398] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552770) [2025-05-15 09:40:30.595406] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552768, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.595417] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552770, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.595427] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552769, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.595457] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221552769, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.595469] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552769, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302030594700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.595504] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=34] mark session id unused(conn.sessid_=3221552769) [2025-05-15 09:40:30.595534] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] connection close(sessid=3221552769, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.596729] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=36] new mysql sessid created(conn.sessid_=3221552771, support_ssl=false) [2025-05-15 09:40:30.596792] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221552771, sess.client_addr_="172.22.0.1:45048") [2025-05-15 09:40:30.596809] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552771, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.596835] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552772, support_ssl=false) [2025-05-15 09:40:30.596848] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552773, support_ssl=false) [2025-05-15 09:40:30.596861] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221552772, sess.client_addr_="172.22.0.1:45058") [2025-05-15 09:40:30.596874] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a6f616030, session_id:3221552772, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.596903] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552773, sess.client_addr_="172.22.0.1:45060") [2025-05-15 09:40:30.596923] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552773, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.597087] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.597123] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.597141] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.597159] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.597148] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.597176] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.597194] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.597202] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.597208] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.597172] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.597221] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=48][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.597222] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.597236] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552772, proxy_sessid=0) [2025-05-15 09:40:30.597238] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=16][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.597254] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=7][errcode=0] server close connection(sessid=3221552772, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.597266] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552772, proxy_sessid=0) [2025-05-15 09:40:30.597262] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.597273] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:30.597261] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.597280] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9080-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552772, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.597284] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552771, proxy_sessid=0) [2025-05-15 09:40:30.597293] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.597303] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.597319] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.597327] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.597325] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=17][errcode=0] server close connection(sessid=3221552771, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.597326] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f616030, session_id:3221552772, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030596814, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.597335] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.597345] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552772, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.597342] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552771, proxy_sessid=0) [2025-05-15 09:40:30.597352] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.597356] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:30.597353] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f616030, session_id:3221552772, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030596814, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.597367] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552772) [2025-05-15 09:40:30.597365] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552773, proxy_sessid=0) [2025-05-15 09:40:30.597368] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9046-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552771, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.597387] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=12][errcode=0] server close connection(sessid=3221552773, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.597381] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552772, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.597400] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552773, proxy_sessid=0) [2025-05-15 09:40:30.597407] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.597415] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552771, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302030596814, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.597420] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9050-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552773, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.597432] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552771, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.597453] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552771, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302030596814, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.597474] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552771) [2025-05-15 09:40:30.597464] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552773, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302030596814, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.597493] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552773, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.597496] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552771, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.597512] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552773, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302030596814, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.597537] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] mark session id unused(conn.sessid_=3221552773) [2025-05-15 09:40:30.597549] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552773, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.599186] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221552774, support_ssl=false) [2025-05-15 09:40:30.599232] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552774, sess.client_addr_="172.22.0.1:45064") [2025-05-15 09:40:30.599250] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552774, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.599430] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.599464] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.599483] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.599510] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=26][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.599518] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.599525] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.599545] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.599560] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552774, proxy_sessid=0) [2025-05-15 09:40:30.599578] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=8][errcode=0] server close connection(sessid=3221552774, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.599591] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552774, proxy_sessid=0) [2025-05-15 09:40:30.599598] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:30.599611] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9081-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552774, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.599661] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552774, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030598925, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.599687] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552774, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.599695] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552774, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302030598925, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.599713] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552774) [2025-05-15 09:40:30.599727] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552774, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.600437] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552775, support_ssl=false) [2025-05-15 09:40:30.600461] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.600496] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552775, sess.client_addr_="172.22.0.1:45076") [2025-05-15 09:40:30.600519] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552775, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.600568] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552776, support_ssl=false) [2025-05-15 09:40:30.600619] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552776, sess.client_addr_="172.22.0.1:45086") [2025-05-15 09:40:30.600635] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552776, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.600698] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.600728] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.600730] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552777, support_ssl=false) [2025-05-15 09:40:30.600762] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=32][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.600777] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552777, sess.client_addr_="172.22.0.1:45096") [2025-05-15 09:40:30.600785] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.600790] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552777, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.600797] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.600806] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.600819] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.600825] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.600840] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552775, proxy_sessid=0) [2025-05-15 09:40:30.600859] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.600869] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=11][errcode=0] server close connection(sessid=3221552775, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.600878] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552775, proxy_sessid=0) [2025-05-15 09:40:30.600874] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.600888] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:30.600890] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.600898] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.600896] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9047-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552775, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.600906] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.600922] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.600937] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552776, proxy_sessid=0) [2025-05-15 09:40:30.600948] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552775, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030599980, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.600954] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=7][errcode=0] server close connection(sessid=3221552776, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.600974] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552775, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.600970] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552776, proxy_sessid=0) [2025-05-15 09:40:30.600973] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:30.600984] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552775, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302030599980, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.600986] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:30.601001] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552775) [2025-05-15 09:40:30.601005] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:30.601004] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.601007] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9051-0-0] [lt=20] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552776, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.601020] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:30.601015] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552775, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.601033] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:30.601028] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13670698, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.601045] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:30.601046] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13670698, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.601055] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:30.601061] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.601055] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552776, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030599980, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.601072] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.601077] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=13670698, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.601081] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552776, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.601085] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552777, proxy_sessid=0) [2025-05-15 09:40:30.601093] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552776, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302030599980, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.601091] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.601112] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=8][errcode=0] server close connection(sessid=3221552777, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.601113] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552776) [2025-05-15 09:40:30.601122] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552777, proxy_sessid=0) [2025-05-15 09:40:30.601117] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.601132] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:30.601133] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552776, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.601138] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.601142] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9082-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552777, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:30.601157] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.601167] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.601188] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.601186] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552777, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302030599980, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:30.601202] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030601200, replica_locations:[]}) [2025-05-15 09:40:30.601208] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552777, proxy_sessid=0, ret=0) [2025-05-15 09:40:30.601224] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.601222] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552777, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302030599980, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:30.601240] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552777) [2025-05-15 09:40:30.601241] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:209, local_retry_times:209, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:30.601263] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.601273] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:30.601277] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12332849841, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.601292] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.601302] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.601274] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221552777, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:30.601297] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.601310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:30.601320] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.601340] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.601345] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:30.601349] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.601360] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.601365] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12332849841, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.601376] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.601393] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.601404] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.601407] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11794491883, cache_obj->added_lc()=false, cache_obj->get_object_id()=15732, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:30.601417] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.601427] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.601441] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.601448] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030601257) [2025-05-15 09:40:30.602576] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.602605] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.602641] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.602659] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.602674] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.602689] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:30.602697] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:30.602713] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:30.602727] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:30.602744] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.602761] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.602770] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.602788] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.602800] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030602799, replica_locations:[]}) [2025-05-15 09:40:30.602819] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.602836] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.602847] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.602867] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:30.602875] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:30.602889] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:30.602909] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:30.602927] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.602935] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.602948] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:30.602968] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:30.602975] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:30.602991] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:30.603007] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:30.603014] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:30.603028] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:30.603037] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:30.603050] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:30.603059] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:30.603079] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:30.603093] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.603102] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:30.603116] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:30.603133] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:30.603146] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=210, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:30.603175] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] will sleep(sleep_us=100000, remain_us=13668507, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=210, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.605370] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.613312] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.613335] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.613357] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.613376] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.613392] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.613403] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.613418] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.613427] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.613441] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302030613285, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:30.613458] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.615494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.618429] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.618455] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.618476] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.619539] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:30.619563] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.619574] WDIAG [CLOG] nonblock_get_leader (ob_location_adapter.cpp:127) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] location adapter nonblock_get_leader failed(ret=-4721, id=1, leader="0.0.0.0:0") [2025-05-15 09:40:30.619591] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.619615] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:30.619623] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:30.619633] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:30.622999] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.623029] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030622993) [2025-05-15 09:40:30.623106] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=78][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302030423093, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.623140] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12332894289, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.623161] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.623187] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.623208] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.623284] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12332722851, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.623308] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.623331] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.625607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.630281] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=74][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:30.630303] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.630314] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.630389] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] Cache replace map node details(ret=0, replace_node_count=0, replace_time=2874, replace_start_pos=5700000, replace_num=100000) [2025-05-15 09:40:30.630412] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=10) [2025-05-15 09:40:30.633286] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-05-15 09:40:30.634811] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4463][T1001_TimerWK3_ReqMemEvict][T1001][YB427F000001-000635291F5C646B-0-0] [lt=34][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.635722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.645838] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.647220] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EAC40-0-0] [lt=42][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302030225997}) [2025-05-15 09:40:30.655953] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.667593] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.677722] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.687860] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.697979] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.700523] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.701322] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.701351] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030701316) [2025-05-15 09:40:30.701368] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302030501260, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.701401] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12332950174, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.701421] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.701467] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=45][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.701503] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12332950174, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.701520] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.703294] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.703317] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13568403, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.703366] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=49][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13568403, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.703388] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.703407] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.703416] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.703434] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.703455] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030703453, replica_locations:[]}) [2025-05-15 09:40:30.703475] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.703507] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:210, local_retry_times:210, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:30.703529] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.703550] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.703563] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:30.703569] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:30.703591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:30.704665] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.704693] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.704726] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.704744] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.704759] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.704783] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.704800] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.704809] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.704824] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.704840] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030704840, replica_locations:[]}) [2025-05-15 09:40:30.704859] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.704873] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.704886] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:30.704899] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:30.704917] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:30.704925] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:30.704945] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:30.704962] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.704971] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:30.704986] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:30.704994] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:30.705007] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:30.705017] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:30.705030] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:30.705038] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:30.705051] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:30.705061] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:30.705069] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:30.705083] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:30.705096] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:30.705110] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:30.705120] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:30.705133] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:30.705145] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:30.705158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=211, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:30.705185] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] will sleep(sleep_us=100000, remain_us=13566496, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=211, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.708096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.709698] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44870239027, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:30.718212] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.718484] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:30.718518] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.718538] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:30.719583] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] LogLoopThread round_cost_time(us)(round_cost_time=137) [2025-05-15 09:40:30.719613] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:30.719630] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:30.719648] WDIAG [PALF] get_leader_from_cache_ (log_sliding_window.cpp:2149) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock_get_leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:30.719665] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.723076] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=12332994680, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.723107] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.723304] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.723344] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030723298) [2025-05-15 09:40:30.723379] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302030523285, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.723409] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12332822187, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.723427] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.728329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.730339] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.738458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.741161] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3954][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=22] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:30.748581] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.752978] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:30.753000] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:30.753016] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:30.753034] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=12478464) [2025-05-15 09:40:30.753069] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=35, size_used=0, mem_used=0) [2025-05-15 09:40:30.753110] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:30.758692] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.765111] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:32, tid:4630}]) [2025-05-15 09:40:30.768805] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.778922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.789047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.799179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.800595] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.801402] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=12333049415, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.801431] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.801457] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.801477] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.801486] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.801516] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1001}, delta=12333049415, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.801530] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.801548] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.801559] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.801571] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.801584] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.801597] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.801605] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030801377) [2025-05-15 09:40:30.805334] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.805362] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13466372, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.805381] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13466372, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.805415] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.805434] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.805444] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.805465] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.805497] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030805495, replica_locations:[]}) [2025-05-15 09:40:30.805520] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.805550] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.806964] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.806988] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.807008] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.807033] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.807050] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.807060] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.807078] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.807090] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030807090, replica_locations:[]}) [2025-05-15 09:40:30.807163] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13464519, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=212, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.809293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.813404] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.818547] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.819419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.819655] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:30.823116] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.823154] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030823110) [2025-05-15 09:40:30.823176] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302030623120, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.823204] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12333094953, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.823227] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.823255] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.823277] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.823286] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.823306] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.823321] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.823329] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030823188) [2025-05-15 09:40:30.823371] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=12332922460, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.823394] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.823417] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:30.823429] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.823443] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.823455] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.823469] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.823477] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030823360) [2025-05-15 09:40:30.826897] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4463][T1001_TimerWK3_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=1] ====== check clog disk timer task ====== [2025-05-15 09:40:30.826932] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4463][T1001_TimerWK3_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=30] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:30.829540] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.830510] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=16] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=9) [2025-05-15 09:40:30.839651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.844278] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5323) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=0] is_in_sync(ret=-4721, palf_id=1, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1747289697927760000, v:0}) [2025-05-15 09:40:30.849782] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.852174] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1747302030852172, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-05-15 09:40:30.854761] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=22] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:30.858054] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=60] PNIO [ratelimit] time: 1747302030858052, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-05-15 09:40:30.859893] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.870005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.880116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.890230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.894971] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-05-15 09:40:30.900341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.900643] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:30.901441] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.901476] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030901434) [2025-05-15 09:40:30.901517] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302030701385, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.901538] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:30.901575] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=12333149891, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.901594] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.901619] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.901633] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.901648] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.901661] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1001}, delta=12333149891, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:30.901677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.901694] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:30.901704] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:30.901717] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:30.901727] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.901740] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:30.901747] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030901557) [2025-05-15 09:40:30.905531] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:30.905539] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:30.907293] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.907316] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13364407, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.907335] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13364407, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.907352] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.907372] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.907389] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.907398] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.907417] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.907432] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030907430, replica_locations:[]}) [2025-05-15 09:40:30.907453] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:30.907481] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:30.908670] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.908693] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.908712] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.908736] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.908752] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.908761] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.908779] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.908791] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030908790, replica_locations:[]}) [2025-05-15 09:40:30.908853] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13362828, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=213, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:30.909982] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:30.910453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.912326] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.917291] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:30.917311] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=18] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-05-15 09:40:30.917324] WDIAG [SQL.PC] run_plan_cache_task (ob_plan_cache.cpp:2766) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] fail to update plan cache memory sys val(ret=-4029) [2025-05-15 09:40:30.917342] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=15] start lib cache evict(tenant_id=1001, mem_hold=2097152, mem_limit=215965365, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:30.917359] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=16] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=2097152, mem_limit=215965365, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:30.917371] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=10] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:30.919876] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=12] Dumping All Cache Objs(alloc_obj_list.count()=0, alloc_obj_list=[]) [2025-05-15 09:40:30.919896] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4461][T1001_TimerWK1_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=19] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:30.920573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.922676] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [4500][T1001_ArcFetche][T1001][YB427F000001-000635291CAC6112-0-0] [lt=38] ObArchiveFetcher is running(thread_index=0) [2025-05-15 09:40:30.923171] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:30.923195] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302030923169, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302030923169, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:30.923259] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=41] get wrs ts(ls_id={id:1}, delta=12333194368, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:30.923281] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.923308] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:30.923330] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.923339] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.923360] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.923378] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:30.923386] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030923237) [2025-05-15 09:40:30.923419] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:30.923448] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302030923413) [2025-05-15 09:40:30.923468] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302030723396, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:30.923518] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12333022931, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:30.923540] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:30.923561] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:30.923578] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:30.923588] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:30.923606] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.923619] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:30.923626] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302030923484) [2025-05-15 09:40:30.928258] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=28][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.930681] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.932430] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4812001768, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:30.932531] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:30.932550] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=20] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4812001768, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:30.934056] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.934078] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.934103] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.934121] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.934140] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.934156] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.934167] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.934184] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.934195] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.934212] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:30.934222] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:30.934236] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:30.934259] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:30.934274] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.934290] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.934310] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.934326] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:30.934336] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:30.934351] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:30.934361] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302030934360, replica_locations:[]}) [2025-05-15 09:40:30.934383] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:30.934428] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:30.934446] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:30.934484] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=38] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:30.934520] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=36][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:30.934545] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=25] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=136, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:30.934573] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=27][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:30.934603] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=27][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:30.934613] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:30.940797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.950914] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.956150] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=30] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:30.957499] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=84] tx gc loop thread is running(MTL_ID()=1) [2025-05-15 09:40:30.957526] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=26] try gc retain ctx [2025-05-15 09:40:30.957533] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] try to retry start_working [2025-05-15 09:40:30.957564] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=13] finish update min start scn(max_decided_scn={val:1747289697727835001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289694525675001, v:0}, keep_alive_scn_:{val:1747289694525675001, v:0}, update_ts_:1747302030957129}) [2025-05-15 09:40:30.958156] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:30.961032] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.965528] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:47, tid:3961}, {errcode:-5627, dropped:86, tid:4602}]) [2025-05-15 09:40:30.968054] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=30] clear dtl interm result cost(us)(clear_cost=2618, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-15 09:40:30.970165] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=14] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:30.970191] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=21] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:30.970210] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=16] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-05-15 09:40:30.970219] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=7] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=76) [2025-05-15 09:40:30.971143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.974192] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4326][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=27] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1ac33de2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:30.978195] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:30.978253] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:30.981258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.991372] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:30.995407] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD4-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.995428] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD4-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.995452] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD4-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.995472] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD4-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.995497] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD4-0-0] [lt=24][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.995514] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:30.995526] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.995541] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.995549] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302030995377, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:30.995566] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.995575] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:30.995659] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:30.996860] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.996882] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:30.996882] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD5-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.996890] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:30.996897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD5-0-0] [lt=14][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:30.996902] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:30.996912] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD5-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:30.996913] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:30.996928] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD5-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:30.996927] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:30.996935] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD5-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:30.996942] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:30.996944] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:30.996955] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:30.996958] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:30.996963] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:30.996971] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302030996873, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:30.996985] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:30.996993] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:30.996990] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:30.996995] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:30.997029] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:30.998320] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:31.000697] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.001484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.001521] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333250307, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.001548] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.001573] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.001594] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.001605] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.001625] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=12333250307, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.001642] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.001654] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.001670] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.001677] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.001694] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.001708] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.001715] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031001501) [2025-05-15 09:40:31.008987] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.009010] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13262714, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.009030] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13262714, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.009047] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.009057] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=13262714, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.009076] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.009097] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.009111] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.009128] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.009138] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.009156] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.009178] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031009176, replica_locations:[]}) [2025-05-15 09:40:31.009200] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.009229] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.009244] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:31.009287] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.009341] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11794899818, cache_obj->added_lc()=false, cache_obj->get_object_id()=15736, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.010677] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.010702] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.010723] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.010734] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.010748] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.010764] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.010783] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.010800] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.010817] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.010826] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.010844] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.010862] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031010861, replica_locations:[]}) [2025-05-15 09:40:31.010935] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13260746, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=214, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.011607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.013508] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.013533] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.013551] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.013572] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.013582] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.013598] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.013609] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.013622] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.013631] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302031013479, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.013648] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.013657] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.013718] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.013732] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.013741] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.013754] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.021716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.023137] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4580][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=0] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:31.023165] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4580][T1002_TimerWK3_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=21] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=75) [2025-05-15 09:40:31.023252] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12333294796, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.023275] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.023302] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.023322] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.023332] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.023350] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.023364] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.023371] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031023237) [2025-05-15 09:40:31.023497] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333122303, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.023523] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.023546] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.023571] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.023580] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.023601] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.023615] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.023622] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031023476) [2025-05-15 09:40:31.030602] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:31.031828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.041943] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.052079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.052632] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.052683] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:31.052693] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC9673-0-0] [lt=39][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302031052662}) [2025-05-15 09:40:31.052714] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:31.052735] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:31.052757] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:31.052777] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=9112, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:31.053128] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.053142] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:31.053157] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:31.056713] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [4460][T1001_TimerWK0_ReplayProcessSt][T1001][Y0-0000000000000000-0-0] [lt=18] get_replay_process success(id={id:1}, submitted_log_size=144320102, unsubmitted_log_size=144320102, replayed_log_size=144320102, unreplayed_log_size=0) [2025-05-15 09:40:31.056755] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [4460][T1001_TimerWK0_ReplayProcessSt][T1001][Y0-0000000000000000-0-0] [lt=24] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=137, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=137, replayed_log_size(MB)=137, last_submitted_log_size(MB)=137, last_replayed_log_size(MB)=137, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-15 09:40:31.058832] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=36] new mysql sessid created(conn.sessid_=3221552788, support_ssl=false) [2025-05-15 09:40:31.058894] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221552788, sess.client_addr_="172.22.0.1:45100") [2025-05-15 09:40:31.058910] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552788, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.059086] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.059127] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.059155] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.059180] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.059193] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.059203] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.059228] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.059244] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552788, proxy_sessid=0) [2025-05-15 09:40:31.059273] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=9][errcode=0] server close connection(sessid=3221552788, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.059284] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552788, proxy_sessid=0) [2025-05-15 09:40:31.059293] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:31.059304] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9048-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552788, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.059349] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552788, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031058581, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.059374] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552788, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.059382] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552788, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031058581, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.059397] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552788) [2025-05-15 09:40:31.059414] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552788, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.062192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.065754] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4076, dropped:3, tid:4542}]) [2025-05-15 09:40:31.065992] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4579][T1002_TimerWK2_Flush][T1002][Y0-0000000000000000-0-0] [lt=15] ====== traversal_flush timer task ====== [2025-05-15 09:40:31.066062] INFO [MDS] flush (mds_table_mgr.cpp:161) [4579][T1002_TimerWK2_Flush][T1002][YB427F000001-000635291F9C665C-0-0] [lt=21] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1a6ea99150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:31.066186] INFO [MDS] flush (mds_table_mgr.cpp:161) [4579][T1002_TimerWK2_Flush][T1002][YB427F000001-000635291F9C665C-0-0] [lt=97] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1a6eb09150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:31.066248] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4579][T1002_TimerWK2_Flush][T1002][YB427F000001-000635291F9C665C-0-0] [lt=60] succeed to traversal_flush(ret=0, ls_cnt=2) [2025-05-15 09:40:31.068335] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:31.068365] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.068382] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:31.068392] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:31.068406] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:31.068496] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=89][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:31.072310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.082426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.092539] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.096338] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552789, support_ssl=false) [2025-05-15 09:40:31.096359] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552790, support_ssl=false) [2025-05-15 09:40:31.096393] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] sm conn init succ(conn.sessid_=3221552789, sess.client_addr_="172.22.0.1:45112") [2025-05-15 09:40:31.096410] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552789, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.096439] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552791, support_ssl=false) [2025-05-15 09:40:31.096465] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221552791, sess.client_addr_="172.22.0.1:45116") [2025-05-15 09:40:31.096477] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552791, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.096499] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552790, sess.client_addr_="172.22.0.1:45130") [2025-05-15 09:40:31.096526] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552790, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.096986] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552792, support_ssl=false) [2025-05-15 09:40:31.097031] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221552792, sess.client_addr_="172.22.0.1:45136") [2025-05-15 09:40:31.097048] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552792, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.097183] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.097247] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=31] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.097286] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.097311] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.097323] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.097345] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.097345] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=22][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.097352] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.097365] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.097378] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097384] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.097393] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.097399] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.097402] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552791, proxy_sessid=0) [2025-05-15 09:40:31.097408] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.097416] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.097418] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.097426] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.097436] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.097440] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097444] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=14][errcode=0] server close connection(sessid=3221552791, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097447] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.097455] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552789, proxy_sessid=0) [2025-05-15 09:40:31.097457] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552791, proxy_sessid=0) [2025-05-15 09:40:31.097463] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.097473] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.097470] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:31.097482] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=12][errcode=0] server close connection(sessid=3221552789, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097502] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552789, proxy_sessid=0) [2025-05-15 09:40:31.097497] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9052-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552791, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.097515] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:31.097510] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=22] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097528] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552790, proxy_sessid=0) [2025-05-15 09:40:31.097522] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9083-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552789, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.097542] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.097556] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.097559] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=12][errcode=0] server close connection(sessid=3221552790, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097563] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.097558] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552791, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.097570] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552790, proxy_sessid=0) [2025-05-15 09:40:31.097574] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.097578] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:31.097584] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552789, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.097593] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097601] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] kill and revert session(conn.sessid_=3221552791, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.097590] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9049-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552790, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.097609] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552792, proxy_sessid=0) [2025-05-15 09:40:31.097611] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221552789, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.097621] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552791, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.097627] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=9][errcode=0] server close connection(sessid=3221552792, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.097637] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552792, proxy_sessid=0) [2025-05-15 09:40:31.097638] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552791) [2025-05-15 09:40:31.097646] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:31.097648] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552790, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.097660] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9053-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552792, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.097658] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552791, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.097673] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552790, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.097686] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552789, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.097687] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552790, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.097704] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552789) [2025-05-15 09:40:31.097708] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552790) [2025-05-15 09:40:31.097715] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552789, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.097735] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221552790, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.097829] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552792, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.097851] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552792, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.097879] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552792, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031096640, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.097899] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552792) [2025-05-15 09:40:31.097910] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552792, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.098461] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552793, support_ssl=false) [2025-05-15 09:40:31.098528] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552793, sess.client_addr_="172.22.0.1:45140") [2025-05-15 09:40:31.098547] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552793, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.098555] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552794, support_ssl=false) [2025-05-15 09:40:31.098606] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552794, sess.client_addr_="172.22.0.1:45152") [2025-05-15 09:40:31.098618] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552794, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.098721] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221552795, support_ssl=false) [2025-05-15 09:40:31.098768] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552795, sess.client_addr_="172.22.0.1:45156") [2025-05-15 09:40:31.098788] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552795, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.098795] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=97] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.098826] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.098829] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.098845] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.098853] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.098863] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.098872] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.098871] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.098885] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.098891] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.098899] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.098904] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.098913] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.098915] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552793, proxy_sessid=0) [2025-05-15 09:40:31.098928] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.098933] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=8][errcode=0] server close connection(sessid=3221552793, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.098940] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552794, proxy_sessid=0) [2025-05-15 09:40:31.098944] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.098944] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552793, proxy_sessid=0) [2025-05-15 09:40:31.098955] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:31.098962] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.098965] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=12][errcode=0] server close connection(sessid=3221552794, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.098963] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9084-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552793, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.098976] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552794, proxy_sessid=0) [2025-05-15 09:40:31.098975] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.098988] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:31.098989] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.098998] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.098995] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904A-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552794, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.099010] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.099020] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552793, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031098756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.099038] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=22] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.099049] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552795, proxy_sessid=0) [2025-05-15 09:40:31.099040] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552794, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031098756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.099050] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221552793, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.099060] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552794, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.099066] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=8][errcode=0] server close connection(sessid=3221552795, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.099068] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552794, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031098756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.099077] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552795, proxy_sessid=0) [2025-05-15 09:40:31.099063] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552793, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031098756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.099081] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552794) [2025-05-15 09:40:31.099086] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:31.099087] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552793) [2025-05-15 09:40:31.099093] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9054-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552795, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.099097] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552794, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.099106] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552793, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.099133] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552795, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031098756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.099153] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552795, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.099176] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552795, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031098756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.099201] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] mark session id unused(conn.sessid_=3221552795) [2025-05-15 09:40:31.099224] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552795, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.100743] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552796, support_ssl=false) [2025-05-15 09:40:31.100766] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.100798] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552796, sess.client_addr_="172.22.0.1:45170") [2025-05-15 09:40:31.100815] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552796, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.100978] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.101003] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.101013] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.101027] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.101038] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.101047] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.101061] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.101072] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552796, proxy_sessid=0) [2025-05-15 09:40:31.101092] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=9][errcode=0] server close connection(sessid=3221552796, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.101102] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552796, proxy_sessid=0) [2025-05-15 09:40:31.101110] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:31.101118] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9085-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552796, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.101158] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552796, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031100875, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.101182] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552796, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.101192] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552796, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031100875, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.101206] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552796) [2025-05-15 09:40:31.101219] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552796, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.101568] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.101596] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031101561) [2025-05-15 09:40:31.101613] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302030901528, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.101656] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12333349681, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.101654] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4318][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=29] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:31.101677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.101704] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.101722] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.101734] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.101748] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12333349681, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.101762] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.101778] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.101786] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.101796] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.101809] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.101818] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.101829] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031101629) [2025-05-15 09:40:31.102032] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552797, support_ssl=false) [2025-05-15 09:40:31.102085] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552797, sess.client_addr_="172.22.0.1:45178") [2025-05-15 09:40:31.102099] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552797, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.102232] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.102207] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552798, support_ssl=false) [2025-05-15 09:40:31.102259] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.102288] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=44] sm conn init succ(conn.sessid_=3221552798, sess.client_addr_="172.22.0.1:45192") [2025-05-15 09:40:31.102284] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.102308] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.102302] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=38] new mysql sessid created(conn.sessid_=3221552799, support_ssl=false) [2025-05-15 09:40:31.102317] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.102310] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552798, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.102325] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.102347] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.102357] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552799, sess.client_addr_="172.22.0.1:45204") [2025-05-15 09:40:31.102366] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552797, proxy_sessid=0) [2025-05-15 09:40:31.102373] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552799, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.102390] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=9][errcode=0] server close connection(sessid=3221552797, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.102400] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552797, proxy_sessid=0) [2025-05-15 09:40:31.102408] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:31.102416] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904B-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552797, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.102467] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552797, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031101933, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.102503] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=33] kill and revert session(conn.sessid_=3221552797, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.102515] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552797, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031101933, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.102530] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552797) [2025-05-15 09:40:31.102545] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552797, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.102542] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.102570] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.102605] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.102605] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=28] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.102621] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.102621] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.102635] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.102637] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.102642] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.102646] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.102648] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.102653] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.102667] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.102661] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.102678] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552798, proxy_sessid=0) [2025-05-15 09:40:31.102680] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552799, proxy_sessid=0) [2025-05-15 09:40:31.102657] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.102701] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=10][errcode=0] server close connection(sessid=3221552799, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.102705] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=15][errcode=0] server close connection(sessid=3221552798, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.102709] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=7][errcode=-4018] get session fail(ret=-4018, sessid=3221552799, proxy_sessid=0) [2025-05-15 09:40:31.102715] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552798, proxy_sessid=0) [2025-05-15 09:40:31.102720] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:31.102725] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:31.102730] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9055-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552799, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.102737] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9086-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552798, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.102781] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552799, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031101933, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.102789] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552798, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031101933, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.102809] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552799, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.102818] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552799, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031101933, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.102820] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221552798, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.102835] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552799) [2025-05-15 09:40:31.102835] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552798, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031101933, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.102858] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552798) [2025-05-15 09:40:31.102852] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552799, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.102878] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552798, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.110262] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:31.111081] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.111106] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13160624, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.111125] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13160624, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.111143] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.111163] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=13160624, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.111181] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.111201] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.111221] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.111241] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.111250] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.111268] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.111286] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031111284, replica_locations:[]}) [2025-05-15 09:40:31.111309] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.111341] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.111357] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:31.111386] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.111448] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11795001923, cache_obj->added_lc()=false, cache_obj->get_object_id()=15737, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.112797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.112803] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.112824] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.112837] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.112854] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.112862] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.112879] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.112898] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.112915] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.112931] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.112940] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.112959] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.112976] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031112976, replica_locations:[]}) [2025-05-15 09:40:31.113048] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13158633, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=215, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.113668] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.113692] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.113714] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.113735] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.113751] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.113761] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.113776] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.113785] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.113799] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302031113641, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.113816] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.113826] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.113901] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.113916] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.113924] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.113937] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.114611] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:31.114632] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:31.114642] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=9] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:31.114662] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=19] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:31.119874] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=30] gc stale ls task succ [2025-05-15 09:40:31.122922] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.123295] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.123326] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031123288) [2025-05-15 09:40:31.123347] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302031023233, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.123385] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12333395247, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.123407] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.123434] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.123454] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.123464] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.123483] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.123505] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.123513] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031123364) [2025-05-15 09:40:31.123538] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.123563] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031123535) [2025-05-15 09:40:31.123580] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302030923484, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.123610] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12333222754, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.123628] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.123648] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.123668] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.123676] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.123693] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.123714] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.123724] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031123598) [2025-05-15 09:40:31.127395] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:31.127421] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-05-15 09:40:31.127431] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:31.127453] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:31.127473] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11794017818, flush_io_finished_ret_:0, flush_io_finished_round_:1505, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:31.133040] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.143154] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.149953] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4292][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=61] ====== check clog disk timer task ====== [2025-05-15 09:40:31.149979] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4292][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:31.153265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.163384] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.173498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.174373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=25] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.174373] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.174386] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.174713] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.176989] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=95] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:31.177028] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=37][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:31.177052] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=21][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:31.177162] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=107][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:31.177188] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=19][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:31.177203] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:31.177219] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:31.177234] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:31.177242] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:31.177256] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=12][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:31.177265] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:31.177282] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=16][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:31.177292] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:31.177307] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=14][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:31.177338] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=15][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:31.177355] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.177369] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.177386] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=15][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:31.177403] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:31.177422] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:31.177438] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=14][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.177462] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:31.177486] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.177520] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=33][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.177538] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=18][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:31.177568] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=19][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:31.177590] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646C-0-0] [lt=22][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.177612] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=19][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:31.177638] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=23][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.177653] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:31.177663] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:31.177677] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302031176752, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.177695] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:31.177705] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.177808] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:31.177827] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=17][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:31.177844] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=16][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:31.177860] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=15][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:31.177872] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=9][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:31.177891] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=15][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:31.177906] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646C-0-0] [lt=15][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:31.180269] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4291][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=20] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:31.183612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.191041] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=33] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:31.193726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.200822] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.201642] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12333450166, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.201673] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.201703] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.201725] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.201735] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.201755] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=12333450166, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.201771] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.201789] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.201799] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.201807] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.201916] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=107][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.201925] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.201938] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031201621) [2025-05-15 09:40:31.203837] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.213191] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.213217] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13058512, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.213238] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=13058512, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.213256] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.213268] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=13058512, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.213285] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.213334] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=46][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.213348] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.213382] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=34][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.213390] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.213409] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.213431] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031213429, replica_locations:[]}) [2025-05-15 09:40:31.213453] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.213482] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.213507] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:31.213544] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.213600] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11795104077, cache_obj->added_lc()=false, cache_obj->get_object_id()=15738, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.213722] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.213747] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=25][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.213765] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.213783] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.213800] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.213811] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.213825] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.213834] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.213849] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302031213696, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.213864] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.213872] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.213946] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.213961] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.213969] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.213984] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.214324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.214873] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.214898] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.214917] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.214936] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.214944] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.214960] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.214980] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.214998] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.215015] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.215024] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.215042] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.215058] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031215058, replica_locations:[]}) [2025-05-15 09:40:31.215130] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=13056552, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=216, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.223373] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12333494608, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.223403] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.223432] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.223454] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.223463] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.223483] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.223506] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.223514] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031223351) [2025-05-15 09:40:31.223589] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333323172, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.223613] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.223637] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.223655] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.223664] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.223681] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.223695] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.223702] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031223576) [2025-05-15 09:40:31.224975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.230699] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:31.230798] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=10][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:31.230823] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=25][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.230835] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=10][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.230898] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=63][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.230916] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.230925] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.230941] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.230959] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.230970] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.230987] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.230997] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.231015] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.231041] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=12] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.231059] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=15] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:31.232710] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:31.232740] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:31.232751] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:31.232769] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:31.232782] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:31.232796] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:31.232814] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:31.232832] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=16][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:31.232850] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=18][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:31.232869] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=18][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:31.232891] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=22][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:31.232912] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=20][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:31.232948] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=35][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:31.232957] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:31.232978] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=15][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:31.232992] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.233001] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.233015] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:31.233025] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:31.233039] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:31.233046] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.233068] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:31.233090] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.233104] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.233111] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:31.233130] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:31.233146] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.233155] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:31.233170] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.233185] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:31.233194] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:31.233206] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302031232528, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.233223] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:31.233230] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.233314] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:31.233330] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=15][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:31.233340] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=9][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:31.233353] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=13][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:31.233362] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=8][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:31.233378] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=14][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:31.233386] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C638F-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:31.235093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.245209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.253127] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.253150] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:31.253159] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:31.253173] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:31.253215] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=39] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-05-15 09:40:31.253256] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:31.255320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.257877] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:31.257910] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-05-15 09:40:31.257925] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:31.257945] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:31.257965] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11794148299, flush_io_finished_ret_:0, flush_io_finished_round_:1505, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:31.264728] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=33] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:31.265431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.275544] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.285653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.295765] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.296580] WDIAG load_file_to_string (utility.h:673) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:31.296604] WDIAG get_ethernet_speed (utility.cpp:575) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:31.296614] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:31.296630] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:31.300881] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.301682] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001) [2025-05-15 09:40:31.301704] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.301727] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031301678) [2025-05-15 09:40:31.301747] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302031101628, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.301782] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12333550244, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.301803] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.301830] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.301846] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.301861] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.301875] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12333550244, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.301891] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.301909] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.301920] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.301933] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.301942] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.301956] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.301964] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031301763) [2025-05-15 09:40:31.305874] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.310534] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:31.313750] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.313773] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.313793] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.313813] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.313829] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.313846] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.313862] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.313871] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.313886] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302031313724, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.313902] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.313911] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.313985] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.314002] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.314010] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.314025] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.314305] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4289][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=24] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:31.314308] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4291][T1_TimerWK2_Flush][T1][Y0-0000000000000000-0-0] [lt=46] ====== traversal_flush timer task ====== [2025-05-15 09:40:31.314309] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4813183052, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:31.314332] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4289][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=21] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=84) [2025-05-15 09:40:31.314372] INFO [MDS] flush (mds_table_mgr.cpp:161) [4291][T1_TimerWK2_Flush][T1][YB427F000001-000635291D0C63BA-0-0] [lt=16] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1aaf36f150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:31.314485] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4291][T1_TimerWK2_Flush][T1][YB427F000001-000635291D0C63BA-0-0] [lt=108] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-15 09:40:31.314613] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=24] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4813183053, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:31.315262] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.315287] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12956438, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.315302] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12956438, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.315318] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.315328] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=12956438, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.315346] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.315365] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.315384] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.315402] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.315411] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.315429] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.315449] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031315447, replica_locations:[]}) [2025-05-15 09:40:31.315470] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.315505] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.315520] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:31.315544] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.315594] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11795206071, cache_obj->added_lc()=false, cache_obj->get_object_id()=15739, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.315984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.316802] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.316825] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.316843] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.316871] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.316879] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.316896] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.316915] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.316932] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.316949] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.316959] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.316971] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.316987] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031316986, replica_locations:[]}) [2025-05-15 09:40:31.317057] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12954624, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=217, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.323417] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.323463] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031323410) [2025-05-15 09:40:31.323482] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302031123363, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.323524] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=12333595670, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.323584] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=55] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.323625] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.323646] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.323640] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.323656] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.323671] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.323671] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031323636) [2025-05-15 09:40:31.323685] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.323686] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302031123598, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.323694] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031323505) [2025-05-15 09:40:31.323717] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333423177, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.323737] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.323760] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.323782] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.323791] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.323809] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.323822] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.323829] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031323705) [2025-05-15 09:40:31.324731] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4289][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=17] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:31.324756] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4289][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=20] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:31.324774] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4289][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=87) [2025-05-15 09:40:31.326108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.336224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.345060] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [5606][T1_TimerWK4_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.346336] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.352771] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:31.352804] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.352864] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=60][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:31.352875] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:31.352889] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:31.352970] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:31.352981] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.352987] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:31.353000] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:31.353007] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:31.353062] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:31.356452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.356851] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:31.356934] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:31.357006] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:31.357070] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:31.357101] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:10084, total:10094) [2025-05-15 09:40:31.357159] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=32] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:31.357211] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=48] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:10084, total:10094) [2025-05-15 09:40:31.357235] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:31.357258] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:31.357277] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:31.357294] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628766][2nd=1893]}) [2025-05-15 09:40:31.366574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.376686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.386797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.395412] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] Garbage Collector is running(seq_=151, gc_interval=10000000) [2025-05-15 09:40:31.395451] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.395495] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=1] gc_check_member_list_ cost time(ret=0, time_us=58) [2025-05-15 09:40:31.395514] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=14] execute_gc cost time(ret=0, time_us=1) [2025-05-15 09:40:31.395564] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=23] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.395872] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.396260] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.396411] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.396514] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.396523] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.396629] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:31.397116] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=9] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2025-05-15 09:40:31.397146] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2025-05-15 09:40:31.397158] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2025-05-15 09:40:31.397176] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2025-05-15 09:40:31.397188] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:31.397202] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:31.397213] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2025-05-15 09:40:31.397228] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:31.397236] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:31.397249] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=12][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:31.397257] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:31.397272] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:31.397282] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:31.397295] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=13][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:31.397315] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:31.397330] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.397347] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=14][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.397356] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:31.397367] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001, ret=-5019) [2025-05-15 09:40:31.397384] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:31.397393] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.397416] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=18][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:31.397440] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.397454] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.397460] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:31.397481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:31.397509] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A8-0-0] [lt=26][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.397519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4482][T1001_GCCollect][T1001][YB427F000001-000635291FEC61A8-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, aret=-5019, ret=-5019) [2025-05-15 09:40:31.397534] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-05-15 09:40:31.397543] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:31.397556] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:31.397565] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302031396882, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-05-15 09:40:31.397581] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:31.397589] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-05-15 09:40:31.397677] WDIAG [SHARE] get_ls_status_ (ob_ls_status_operator.cpp:1062) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001, tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-05-15 09:40:31.397696] WDIAG [SHARE] get_ls_status_info (ob_ls_status_operator.cpp:626) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] failed to get ls status(ret=-5019, ret="OB_TABLE_NOT_EXIST", id={id:1}, tenant_id=1001) [2025-05-15 09:40:31.397707] INFO [CLOG] get_ls_status_from_table (ob_garbage_collector.cpp:1481) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] failed to get ls status info from table(ret=-5019, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:31.397721] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1766) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] failed to get ls status from table(ret=-5019, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:31.397730] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1666) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=0] get_ls_status_ failed(tmp_ret=-5019, ls->get_ls_id()={id:1}) [2025-05-15 09:40:31.397746] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13] execute_gc cost time(ret=0, time_us=0) [2025-05-15 09:40:31.397755] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyHandler start process [2025-05-15 09:40:31.397769] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-15 09:40:31.397784] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=8] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-15 09:40:31.397894] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.400937] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.401764] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333650532, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.401789] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.401816] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.401837] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.401846] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.401861] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12333650532, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.401884] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.401901] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.401912] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.401924] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.401935] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.401949] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.401957] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031401743) [2025-05-15 09:40:31.408011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.413671] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.413693] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.413707] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.413724] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.413733] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.413750] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.413770] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.413788] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.413798] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302031413657, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.413815] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.413822] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.413872] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.413886] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.413895] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.413908] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.417186] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.417210] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12854512, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.417230] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12854512, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.417246] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.417256] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=12854512, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.417273] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.417292] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.417311] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.417328] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.417337] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.417355] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.417373] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031417372, replica_locations:[]}) [2025-05-15 09:40:31.417394] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.417421] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.417436] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:31.417476] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.417531] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11795308010, cache_obj->added_lc()=false, cache_obj->get_object_id()=15740, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.418125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.418652] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.418675] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.418693] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.418711] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.418719] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.418735] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.418747] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.418764] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.418781] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.418792] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.418809] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.418820] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031418819, replica_locations:[]}) [2025-05-15 09:40:31.418885] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12852797, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=218, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.423498] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333695036, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.423528] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.423555] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.423576] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.423585] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.423604] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.423618] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.423626] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031423469) [2025-05-15 09:40:31.423709] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=12333522543, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.423736] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.423761] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.423779] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.423787] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.423797] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.423810] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.423819] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031423695) [2025-05-15 09:40:31.425961] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=21] ====== checkpoint timer task ====== [2025-05-15 09:40:31.426629] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=20] succ to get rec scn(*this={this:0x7f1ab0204030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300517859523, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1303}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.426674] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=44] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a96ba2080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a96ba2080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.426738] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=35] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1aaf36f150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:31.426830] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=90] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1aaf36e1b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1aaf379010, memtable_head:0, memtable_tail:1, t3m:0x7f1ac66cc030, tables:[0x7f1a96b92080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1aaf37ad10, ls_tablet_svr:0x7f1aaf36e190}, min_rec_scn={val:1747274404592930000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:31.426867] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=31] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-15 09:40:31.426888] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] get rec log scn(service_type_=0, rec_log_ts={val:1747289692924660001, v:0}) [2025-05-15 09:40:31.426903] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=16] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.426912] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.426931] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:1747277617103769000, v:0}, flushed_scn_={val:1746499972147257000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:1747277617937058000, v:0}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:31.426957] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=22] rec_scn of ObLockMemtable is (rec_scn_={val:1747277617103769000, v:0}, flushed_scn_={val:1746499972147257000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:1747277617937058000, v:0}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:31.426977] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274404592930000, v:0}, checkpoint_scn_in_ls_meta={val:1747274404592930000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:31.426996] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2818400256}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2818400256}, lsn={lsn:2818400256}) [2025-05-15 09:40:31.427027] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=30] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2818400256}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:31.427038] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-15 09:40:31.428239] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.430788] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=6) [2025-05-15 09:40:31.438356] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.448467] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.455986] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:133) [3956][TimerWK3_ServerTracerTim][T0][YB427F000001-00063529204C618A-0-0] [lt=8][errcode=-4002] invalid argument, empty server list(ret=-4002) [2025-05-15 09:40:31.456009] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:378) [3956][TimerWK3_ServerTracerTim][T0][YB427F000001-00063529204C618A-0-0] [lt=22][errcode=-4002] refresh sever list failed(ret=-4002) [2025-05-15 09:40:31.456016] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:252) [3956][TimerWK3_ServerTracerTim][T0][YB427F000001-00063529204C618A-0-0] [lt=8][errcode=-4002] refresh alive server list failed(ret=-4002) [2025-05-15 09:40:31.458590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.466562] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:364, tid:4290}]) [2025-05-15 09:40:31.468705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.478818] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.487614] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:31.487673] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:31.488933] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.497044] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.497079] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.497089] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:31.497105] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:31.497114] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:31.497130] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:31.497151] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:31.497164] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:31.497202] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:31.497209] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:31.497229] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.497249] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:31.497275] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:31.497291] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:31.497299] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:31.497314] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:31.497326] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:31.499047] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.501013] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.501799] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.501831] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031501792) [2025-05-15 09:40:31.501864] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302031301762, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.501898] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12333749655, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.501920] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.501946] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.501964] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.501973] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.501992] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12333750712, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.502008] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.502026] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.502036] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.502049] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.502059] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.502073] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.502081] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031501882) [2025-05-15 09:40:31.508657] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=21] ====== checkpoint timer task ====== [2025-05-15 09:40:31.509162] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.509285] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=18] succ to get rec scn(*this={this:0x7f1a69e04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300524716821, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:726}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.509333] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=48] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a690fa080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a690fa080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.509398] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=35] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1a6ea99150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:31.509498] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=86] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1a6ea981b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1a6eaa3010, memtable_head:0, memtable_tail:1, t3m:0x7f1a8f8ea030, tables:[0x7f1a690ea080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1a6eaa4d10, ls_tablet_svr:0x7f1a6ea98190}, min_rec_scn={val:1747274406063927000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:31.509537] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=32] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1747274406063927000, v:0}) [2025-05-15 09:40:31.509559] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=20] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.509575] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=16] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.509590] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=7] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:31.509616] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=22] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:31.509637] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274406063927000, v:0}, checkpoint_scn_in_ls_meta={val:1747274406063927000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:31.509663] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:872361984}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:872361984}, lsn={lsn:872361984}) [2025-05-15 09:40:31.509693] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=29] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:872361984}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:31.510288] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x7f1a69e76030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300524741888, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:5}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.510316] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=29] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a690fb390, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a690fb390, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:31.510352] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=17] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1a6eb09150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:31.510413] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=60] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-05-15 09:40:31.510431] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1747274672721723000, v:0}) [2025-05-15 09:40:31.510445] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-05-15 09:40:31.510465] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=19] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-05-15 09:40:31.510485] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=18] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747289697751194001, v:0}, checkpoint_scn_in_ls_meta={val:1747289697751194001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-05-15 09:40:31.510514] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:738152448}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1746497901775524000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:738152448}, lsn={lsn:738152448}) [2025-05-15 09:40:31.510535] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=21] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:738152448}, ls->get_ls_id()={id:1001}) [2025-05-15 09:40:31.510543] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4580][T1002_TimerWK3_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=2) [2025-05-15 09:40:31.510796] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:31.513817] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.513841] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.513860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.513879] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.513894] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.513904] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.513920] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.513929] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.513944] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302031513794, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.513964] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.513977] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.514043] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.514057] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.514066] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.514080] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.518943] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.518967] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.518981] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.519029] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.519053] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12752673, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.519073] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12752673, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.519090] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.519100] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=12752673, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.519119] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.519139] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.519157] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.519175] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.519186] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.519204] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.519225] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031519223, replica_locations:[]}) [2025-05-15 09:40:31.519245] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.519268] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:218, local_retry_times:218, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:31.519277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.519296] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.519314] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:31.519330] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:31.519344] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:31.519351] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:31.519368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:31.519385] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.519435] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11795409912, cache_obj->added_lc()=false, cache_obj->get_object_id()=15741, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.520100] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:31.520122] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.520134] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.520155] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302031519894, register_time_us=-1, curr_time_us=1747302031520152) [2025-05-15 09:40:31.520177] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:31.520186] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.520201] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:31.520222] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:31.520236] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:31.520244] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:31.520254] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8] re_register_parent reason(palf_id=1001, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302031519894, register_time_us=-1, curr_time_us=1747302031520253) [2025-05-15 09:40:31.520270] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:31.520278] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:31.520293] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1001, self="127.0.0.1:2882") [2025-05-15 09:40:31.520647] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.520676] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.520713] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.520731] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.520749] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.520766] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.520774] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.520790] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.520803] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.520820] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.520838] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.520853] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.520866] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.520883] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031520882, replica_locations:[]}) [2025-05-15 09:40:31.520902] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.520919] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.520935] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.520954] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:31.520963] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:31.520977] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:31.520998] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:31.521017] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:31.521024] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:31.521038] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:31.521046] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:31.521059] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:31.521070] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:31.521086] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:31.521094] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:31.521108] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:31.521116] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:31.521129] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:31.521139] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:31.521158] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:31.521172] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.521182] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:31.521196] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:31.521213] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:31.521221] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=219, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.521256] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24] will sleep(sleep_us=100000, remain_us=12750425, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=219, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.523532] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.523564] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031523526) [2025-05-15 09:40:31.523590] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302031323504, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.523625] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12333795229, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.523646] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.523671] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.523690] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.523699] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.523722] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.523737] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.523744] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031523610) [2025-05-15 09:40:31.523777] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.523812] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031523770) [2025-05-15 09:40:31.523831] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302031323700, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.523859] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12333622736, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.523878] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.523898] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.523917] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.523926] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.523944] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.523951] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.523964] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031523849) [2025-05-15 09:40:31.529389] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.530851] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:31.530877] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.530889] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.538341] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=22] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:31.539505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.540950] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [4064][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=27] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-05-15 09:40:31.541666] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=22] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:31.549619] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.553651] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.553690] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=37][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.553718] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:31.553764] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:31.553778] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:31.553803] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] refresh gts failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:31.553819] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:31.553828] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:31.553842] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.553863] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:31.553889] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:31.553906] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:31.553915] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:31.553928] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:31.553938] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:31.553956] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:31.554327] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:31.554344] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.554359] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.554372] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:31.554380] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:31.554394] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:31.554400] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:31.554413] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.554423] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:31.560007] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=33] new mysql sessid created(conn.sessid_=3221552808, support_ssl=false) [2025-05-15 09:40:31.560099] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=45] sm conn init succ(conn.sessid_=3221552808, sess.client_addr_="172.22.0.1:45218") [2025-05-15 09:40:31.560121] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552808, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.560344] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.560391] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.560427] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.560462] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=33][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.560476] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.560495] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.560520] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.560537] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552808, proxy_sessid=0) [2025-05-15 09:40:31.560566] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=11][errcode=0] server close connection(sessid=3221552808, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.560577] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552808, proxy_sessid=0) [2025-05-15 09:40:31.560586] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:31.560600] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904C-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552808, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.560664] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552808, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031560067, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.560694] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221552808, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.560710] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552808, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031560067, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.560731] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552808) [2025-05-15 09:40:31.560758] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552808, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.561862] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.563390] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C665E-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.563415] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C665E-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.563435] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C665E-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.563454] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C665E-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.563464] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.563480] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.563511] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=30][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.563523] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.563537] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302031563366, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.563555] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.563564] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:31.563641] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=15][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:31.563659] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=17][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:31.563669] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=10][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:31.563683] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=14][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:31.563693] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=7][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:31.563708] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=14][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:31.563717] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C665E-0-0] [lt=8][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:31.569781] INFO pn_ratelimit (group.c:653) [3955][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=30] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-15 09:40:31.571986] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.582101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.592222] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.598695] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552810, support_ssl=false) [2025-05-15 09:40:31.598744] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552810, sess.client_addr_="172.22.0.1:45230") [2025-05-15 09:40:31.598761] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552810, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.598916] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552811, support_ssl=false) [2025-05-15 09:40:31.598961] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.598993] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.599015] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=43] sm conn init succ(conn.sessid_=3221552811, sess.client_addr_="172.22.0.1:45236") [2025-05-15 09:40:31.599032] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552811, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.599031] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.599062] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=30][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.599074] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.599088] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.599083] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552812, support_ssl=false) [2025-05-15 09:40:31.599113] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599138] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552810, proxy_sessid=0) [2025-05-15 09:40:31.599144] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] sm conn init succ(conn.sessid_=3221552812, sess.client_addr_="172.22.0.1:45246") [2025-05-15 09:40:31.599161] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552812, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.599184] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=17][errcode=0] server close connection(sessid=3221552810, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599200] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552810, proxy_sessid=0) [2025-05-15 09:40:31.599209] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:31.599223] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552813, support_ssl=false) [2025-05-15 09:40:31.599226] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9056-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552810, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.599264] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221552813, sess.client_addr_="172.22.0.1:45256") [2025-05-15 09:40:31.599284] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552810, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302031598109, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.599284] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552813, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.599309] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552810, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.599318] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552810, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302031598109, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.599341] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552810) [2025-05-15 09:40:31.599337] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.599353] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552810, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.599373] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.599382] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.599402] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.599407] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.599427] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.599436] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.599444] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.599448] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.599437] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.599468] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599471] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.599465] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.599480] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552811, proxy_sessid=0) [2025-05-15 09:40:31.599480] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.599485] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.599502] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=21][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.599510] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.599512] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=19][errcode=0] server close connection(sessid=3221552811, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599519] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.599526] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.599525] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552811, proxy_sessid=0) [2025-05-15 09:40:31.599527] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599536] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:31.599540] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599553] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552813, proxy_sessid=0) [2025-05-15 09:40:31.599548] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552812, proxy_sessid=0) [2025-05-15 09:40:31.599552] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9087-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552811, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.599570] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=7][errcode=0] server close connection(sessid=3221552813, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599582] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552813, proxy_sessid=0) [2025-05-15 09:40:31.599590] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:31.599589] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=16][errcode=0] server close connection(sessid=3221552812, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.599600] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9057-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552813, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.599620] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=31][errcode=-4018] get session fail(ret=-4018, sessid=3221552812, proxy_sessid=0) [2025-05-15 09:40:31.599631] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:31.599642] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904D-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552812, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.599681] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552811, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031599169, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.599718] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552812, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031599169, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.599759] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=38] kill and revert session(conn.sessid_=3221552812, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.599752] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=45][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552813, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031599169, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.599789] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221552811, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.599782] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552812, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031599169, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.599812] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] mark session id unused(conn.sessid_=3221552812) [2025-05-15 09:40:31.599814] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552813, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.599828] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552811, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031599169, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.599861] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] mark session id unused(conn.sessid_=3221552811) [2025-05-15 09:40:31.599839] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552812, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.599888] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552811, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.599932] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=41] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552813, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031599169, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.599970] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=37] mark session id unused(conn.sessid_=3221552813) [2025-05-15 09:40:31.599999] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] connection close(sessid=3221552813, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.600116] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552814, support_ssl=false) [2025-05-15 09:40:31.600158] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552815, support_ssl=false) [2025-05-15 09:40:31.600197] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552815, sess.client_addr_="172.22.0.1:45270") [2025-05-15 09:40:31.600193] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] sm conn init succ(conn.sessid_=3221552814, sess.client_addr_="172.22.0.1:45266") [2025-05-15 09:40:31.600211] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552815, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.600220] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552814, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.600391] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.600399] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.600396] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=51] new mysql sessid created(conn.sessid_=3221552816, support_ssl=false) [2025-05-15 09:40:31.600420] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.600442] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.600440] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.600451] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.600452] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552816, sess.client_addr_="172.22.0.1:45284") [2025-05-15 09:40:31.600460] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.600465] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.600469] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.600473] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.600477] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.600469] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552816, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.600481] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.600503] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.600511] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=24] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.600518] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552815, proxy_sessid=0) [2025-05-15 09:40:31.600526] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552814, proxy_sessid=0) [2025-05-15 09:40:31.600547] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=11][errcode=0] server close connection(sessid=3221552814, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.600544] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=14][errcode=0] server close connection(sessid=3221552815, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.600555] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552815, proxy_sessid=0) [2025-05-15 09:40:31.600555] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=7][errcode=-4018] get session fail(ret=-4018, sessid=3221552814, proxy_sessid=0) [2025-05-15 09:40:31.600563] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:31.600567] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:31.600570] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9088-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552815, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.600574] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9058-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552814, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.600622] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552815, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302031600347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.600625] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552814, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031600347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.600646] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552815, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.600640] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.600649] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552814, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.600653] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552815, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302031600347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.600666] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221552815) [2025-05-15 09:40:31.600659] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552814, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302031600347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.600675] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.600677] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552814) [2025-05-15 09:40:31.600679] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552815, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.600694] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552814, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.600696] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=43][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.600721] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.600733] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.600747] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.600764] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.600783] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552816, proxy_sessid=0) [2025-05-15 09:40:31.600808] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=9][errcode=0] server close connection(sessid=3221552816, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.600823] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552816, proxy_sessid=0) [2025-05-15 09:40:31.600831] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:31.600846] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904E-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552816, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.600853] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.600879] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.600898] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] failed to get percent [2025-05-15 09:40:31.600895] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552816, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031600347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.600907] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] failed to calculate global bound size(ret=-5627, ids[i]=1001) [2025-05-15 09:40:31.600917] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552816, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.600931] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552816, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302031600347, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.600939] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.600951] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552816) [2025-05-15 09:40:31.600956] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.600965] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:31.600974] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:31.600980] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:31.600968] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552816, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.600987] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] failed to get percent [2025-05-15 09:40:31.600995] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] failed to calculate global bound size(ret=-5157, ids[i]=1002) [2025-05-15 09:40:31.601063] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.601882] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333850224, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.601914] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.601945] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.601967] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.601978] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.601992] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12333850224, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.602009] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.602028] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.602034] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552817, support_ssl=false) [2025-05-15 09:40:31.602040] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.602052] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.602062] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.602077] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.602085] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031601857) [2025-05-15 09:40:31.602084] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552817, sess.client_addr_="172.22.0.1:45294") [2025-05-15 09:40:31.602102] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552817, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.602254] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.602286] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.602296] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.602312] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.602326] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.602334] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.602355] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.602343] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.602372] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552817, proxy_sessid=0) [2025-05-15 09:40:31.602389] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=8][errcode=0] server close connection(sessid=3221552817, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.602403] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552817, proxy_sessid=0) [2025-05-15 09:40:31.602411] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:31.602424] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9089-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552817, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.602467] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552817, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302031601418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.602501] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] kill and revert session(conn.sessid_=3221552817, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.602511] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552817, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302031601418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.602531] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552817) [2025-05-15 09:40:31.602542] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552817, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.603429] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552818, support_ssl=false) [2025-05-15 09:40:31.603500] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552818, sess.client_addr_="172.22.0.1:45298") [2025-05-15 09:40:31.603523] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221552818, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.603725] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.603749] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552819, support_ssl=false) [2025-05-15 09:40:31.603767] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.603778] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.603792] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.603799] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.603801] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552819, sess.client_addr_="172.22.0.1:45302") [2025-05-15 09:40:31.603807] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.603814] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552819, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.603827] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.603840] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552820, support_ssl=false) [2025-05-15 09:40:31.603845] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552818, proxy_sessid=0) [2025-05-15 09:40:31.603871] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221552820, sess.client_addr_="172.22.0.1:45316") [2025-05-15 09:40:31.603873] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=16][errcode=0] server close connection(sessid=3221552818, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.603886] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552818, proxy_sessid=0) [2025-05-15 09:40:31.603885] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552820, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.603902] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=16][errcode=-4016] session is null [2025-05-15 09:40:31.603929] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9059-0-0] [lt=27] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552818, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.603950] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.603980] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.603981] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221552818, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031603502, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.604001] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.604016] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.604025] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.604020] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221552818, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.604035] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.604035] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.604037] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221552818, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302031603502, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.604056] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.604053] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552818) [2025-05-15 09:40:31.604057] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.604069] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552819, proxy_sessid=0) [2025-05-15 09:40:31.604073] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.604075] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552818, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.604090] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.604098] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.604099] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=18][errcode=0] server close connection(sessid=3221552819, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.604107] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.604111] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552819, proxy_sessid=0) [2025-05-15 09:40:31.604117] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:31.604122] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.604124] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC904F-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552819, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.604140] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552820, proxy_sessid=0) [2025-05-15 09:40:31.604159] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=8][errcode=0] server close connection(sessid=3221552820, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.604173] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552820, proxy_sessid=0) [2025-05-15 09:40:31.604180] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:31.604175] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552819, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031603502, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.604203] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221552819, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.604199] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908A-0-0] [lt=18] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552820, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.604217] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552819, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031603502, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.604231] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552819) [2025-05-15 09:40:31.604247] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552819, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.604282] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552820, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031603502, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.604301] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552820, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.604354] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552820, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302031603502, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.604371] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552820) [2025-05-15 09:40:31.604384] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552820, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.612476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.613947] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.613970] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.613990] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.614012] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.614022] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.614039] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.614050] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.614065] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.614073] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302031613920, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.614090] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.614099] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.614173] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.614188] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.614196] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.614210] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.619000] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.619027] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.619117] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=88][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.619562] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [4090][Blacklist][T0][Y0-0000000000000000-0-0] [lt=51] blacklist_loop exec finished(cost_time=18, is_enabled=true, send_cnt=0) [2025-05-15 09:40:31.620165] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:31.620196] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.620217] WDIAG [CLOG] nonblock_get_leader (ob_location_adapter.cpp:127) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] location adapter nonblock_get_leader failed(ret=-4721, id=1, leader="0.0.0.0:0") [2025-05-15 09:40:31.620229] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.620250] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:31.620265] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:31.620274] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:31.621411] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.621444] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12650298, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.621464] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12650298, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.621483] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.621504] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=12650298, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.621524] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.621540] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.621562] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.621582] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.621591] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.621612] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.621636] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031621633, replica_locations:[]}) [2025-05-15 09:40:31.621659] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.621685] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:219, local_retry_times:219, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:31.621712] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.621727] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:31.621742] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:31.621756] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:31.621764] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:31.621788] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:31.621804] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.621861] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11795512335, cache_obj->added_lc()=false, cache_obj->get_object_id()=15742, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.622611] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.623206] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.623238] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=30][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.623278] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.623298] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.623316] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.623334] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:31.623342] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:31.623359] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:31.623373] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:31.623391] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.623405] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.623420] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.623432] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.623449] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031623449, replica_locations:[]}) [2025-05-15 09:40:31.623470] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.623496] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.623514] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.623534] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:31.623543] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:31.623558] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:31.623581] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:31.623600] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:31.623608] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:31.623619] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:31.623615] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12333895750, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.623634] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:31.623643] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:31.623644] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.623655] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:31.623670] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:31.623679] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:31.623673] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.623688] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:31.623695] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.623701] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:31.623712] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:31.623712] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.623722] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:31.623727] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.623737] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:31.623740] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.623753] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.623752] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031623589) [2025-05-15 09:40:31.623762] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:31.623779] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:31.623799] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:31.623807] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=220, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.623838] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] will sleep(sleep_us=100000, remain_us=12647844, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=220, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.623844] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12333723257, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.623861] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.623885] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.623897] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.623912] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.623925] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.623939] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.623947] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031623830) [2025-05-15 09:40:31.630881] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:31.630906] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:31.630928] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.630947] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.632728] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.642846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.647656] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4045][LSUserTblUp1][T0][YB427F000001-000635291EED0EB8-0-0] [lt=35][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001, task={tenant_id:1002, ls_id:{id:1001}, add_timestamp:1747300526348819}) [2025-05-15 09:40:31.652963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.663080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.674726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.684846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.694962] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.701118] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.701925] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.701952] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031701918) [2025-05-15 09:40:31.701974] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302031501881, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.702014] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12333950667, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.702036] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.702065] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.702087] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.702096] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.702117] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=12333950667, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.702135] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.702147] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.702163] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.702171] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.702189] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.702204] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.702211] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031701994) [2025-05-15 09:40:31.705073] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.711058] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:31.713961] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.713985] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.714005] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.714028] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.714045] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.714055] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.714065] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.714081] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302031713935, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.714099] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.714107] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:31.714180] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:31.714195] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:31.714204] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:31.714219] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:31.715190] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.719058] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.719081] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.719094] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.720220] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:31.720245] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:31.720257] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:31.720270] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:31.720286] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:31.720295] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:31.721960] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=23] table not exist(tenant_id=1, database_id=201001, table_name=__all_mview, table_name.ptr()="data_size:11, data:5F5F616C6C5F6D76696577", ret=-5019) [2025-05-15 09:40:31.722004] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=41][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_mview, ret=-5019) [2025-05-15 09:40:31.722027] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=21][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_mview, db_name=oceanbase) [2025-05-15 09:40:31.722048] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=20][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_mview) [2025-05-15 09:40:31.722062] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:31.722078] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=16][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:31.722091] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_mview' doesn't exist [2025-05-15 09:40:31.722105] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=12][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:31.722114] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:31.722127] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=12][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:31.722136] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:31.722152] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=16][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:31.722162] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:31.722176] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=13][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:31.722195] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:31.722209] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.722221] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.722237] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=14][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:31.722246] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4, ret=-5019) [2025-05-15 09:40:31.722263] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:31.722271] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.722296] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:31.722321] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.722334] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.722341] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:31.722358] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:31.722375] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.722393] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BB-0-0] [lt=16][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5019, ret=-5019) [2025-05-15 09:40:31.722404] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:31.722419] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:31.722427] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:31.722441] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302031721697, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:31.722458] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:31.723652] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.723684] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031723645) [2025-05-15 09:40:31.723712] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302031523606, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.723749] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12333995185, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.723771] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.723799] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.723896] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.723919] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031723892) [2025-05-15 09:40:31.723936] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302031523849, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.723966] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=12333822692, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.723977] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.723986] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.723996] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12547726, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.724007] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.724013] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12547726, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.724043] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.724063] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.724073] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.724093] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.724113] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031724112, replica_locations:[]}) [2025-05-15 09:40:31.724137] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.724159] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:220, local_retry_times:220, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:31.724184] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.724207] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:31.724221] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:31.724227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:31.724266] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:31.725306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.725441] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.725471] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.725517] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.725537] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.725555] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.725575] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.725592] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.725601] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.725620] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.725638] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031725638, replica_locations:[]}) [2025-05-15 09:40:31.725659] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.725676] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:31.725686] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:31.725706] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:31.725714] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:31.725729] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:31.725750] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:31.725768] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:31.725777] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:31.725793] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:31.725800] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:31.725814] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:31.725824] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:31.725839] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:31.725847] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:31.725856] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:31.725871] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:31.725879] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:31.725894] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:31.725906] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:31.725922] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.725931] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:31.725947] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:31.725957] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:31.725971] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=221, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.725999] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] will sleep(sleep_us=100000, remain_us=12545682, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=221, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.730964] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.735420] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.741251] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3954][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=24] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:31.745534] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.753307] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.753333] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:31.753343] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:31.753364] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:31.753375] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:31.753417] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:31.755644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.765758] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.775873] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.785989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.796104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.801171] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.801978] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:31.801998] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302031801976, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302031801976, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:31.802053] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=38] get wrs ts(ls_id={id:1}, delta=12334050079, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.802075] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.802126] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.802153] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12334050079, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.802170] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.806214] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.813949] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.813973] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.813989] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.814008] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.814028] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.816326] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.819114] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=41][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.819713] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:31.820280] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.823717] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12334095664, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.823742] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.823965] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12333923171, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.823988] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.826122] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.826148] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12445575, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.826166] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12445575, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.826194] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.826213] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.826222] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.826240] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.826253] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031826252, replica_locations:[]}) [2025-05-15 09:40:31.826274] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.826294] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.826436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.827382] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.827406] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.827426] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.827453] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.827470] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.827479] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.827511] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.827522] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031827521, replica_locations:[]}) [2025-05-15 09:40:31.827587] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12444095, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=222, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.830973] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:31.836557] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.846669] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.853484] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4624][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=25] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1a903de2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:31.853841] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.853843] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4291][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=14] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:31.855884] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4243][T1_Occam][T1][YB427F000001-000635291B4C65CB-0-0] [lt=16] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.855929] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4243][T1_Occam][T1][YB427F000001-000635291B4C65CD-0-0] [lt=20] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.856781] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.856985] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:31.857016] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:31.857035] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=18][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:31.857053] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:31.857064] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:31.857074] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:31.857092] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=13][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:31.857107] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=14][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:31.857115] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:31.857129] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=13][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:31.857137] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:31.857152] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=13][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:31.857160] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:31.857174] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=13][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:31.857191] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:31.857206] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.857216] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:31.857230] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:31.857238] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:31.857253] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:31.857260] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:31.857281] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:31.857303] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.857311] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:31.857325] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:31.857339] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:31.857356] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.857365] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4247][T1_Occam][T1][YB427F000001-000635291C6C6240-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:31.857381] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:31.857390] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:31.857406] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:31.857414] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302031856821, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:31.857431] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:31.857440] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f1abce563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:31.857463] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:31.857553] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1abce563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:31.857573] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:31.857583] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:31.857599] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:31.857639] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=33] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:31.859791] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1747302031859789, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-05-15 09:40:31.865669] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1747302031865667, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-05-15 09:40:31.866894] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.867302] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:33, tid:4630}]) [2025-05-15 09:40:31.877004] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.887121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.897232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.901228] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:31.902043] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.902069] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031902036) [2025-05-15 09:40:31.902083] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302031701993, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.902113] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:31.902152] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=12334150496, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.902174] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.902199] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.902220] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.902230] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.902251] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=12334150496, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:31.902268] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.902286] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:31.902298] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:31.902312] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:31.902322] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.902338] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:31.902346] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031902133) [2025-05-15 09:40:31.907346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.908157] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:31.908155] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=39] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:31.911312] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:31.912538] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.912567] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5323) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=0] is_in_sync(ret=-4721, palf_id=1, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1747289698016024000, v:0}) [2025-05-15 09:40:31.917457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.923769] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.923811] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031923763) [2025-05-15 09:40:31.923916] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=106][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302031723728, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.923937] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:31.924013] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:31.924048] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302031924010) [2025-05-15 09:40:31.923973] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] get_min_unreplayed_log_info(lsn={lsn:2944224391}, scn={val:1747289697727835002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300530150701, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:1686016, ret_code_:0}}, next_to_submit_lsn_:{lsn:2944224391}, next_to_submit_scn_:{val:1747289697727835002, v:0}, base_lsn_:{lsn:2818400256}, base_scn_:{val:1747274404592930000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69205000, next_round_pread_size:3678208, curr_read_pos:3609237, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3609237, log_storage_:{start_lsn:{lsn:2940615154}, end_lsn:{lsn:2944224391}, read_buf:{buf_len_:3747840, buf_:0x7f1a69205000}, block_size:67104768, log_storage_:0x7f1ab0f4fbf0, log_storage_:{log_tail:{lsn:2944224980}, readable_log_tail:{lsn:2944224980}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:43, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:0, max_block_id:44, curr_writable_block_id:43}, logical_block_size_:67104768, curr_block_writable_size_:8384812, block_header_serialize_buf_:0x7f1ab0f508b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697727835001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697727835001, v:0}, data_checksum:2923900223, flag:13900}}, init_mode_version:0, accumulate_checksum:2881944827, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697727835001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:64, read_io_size:126980096, read_disk_cost_ts:1613509, start_lsn:{lsn:2818400256}}}}}}) [2025-05-15 09:40:31.924067] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302031723955, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:31.924072] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97] get wrs ts(ls_id={id:1}, delta=12334195943, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:31.924088] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12334023450, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:31.924088] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.924109] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:31.924110] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:31.924121] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.924129] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:31.924134] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.924148] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:31.924152] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.924157] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:31.924164] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:31.924174] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.924176] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031923928) [2025-05-15 09:40:31.924186] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:31.924202] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302031924077) [2025-05-15 09:40:31.927573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.927729] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.927758] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12343973, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.927779] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12343973, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.927798] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.927822] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.927840] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.927849] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.927869] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.927891] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031927889, replica_locations:[]}) [2025-05-15 09:40:31.927913] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:31.927940] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:31.929216] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:31.929241] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.929259] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:31.929286] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.929304] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:31.929313] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:31.929331] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:31.929342] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302031929341, replica_locations:[]}) [2025-05-15 09:40:31.929412] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12342269, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=223, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:31.931093] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", [SlidingWindow]={palf_id:1, self:"127.0.0.1:2882", lsn_allocator:{max_log_id:8886935, max_lsn:5311387831, max_scn:{val:1747289698016024000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:5311387238}, reuse_lsn:{lsn:5311387831}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:5311387831}}, last_submit_lsn:{lsn:5311387238}, last_submit_end_lsn:{lsn:5311387831}, last_submit_log_id:8886935, last_submit_log_pid:4, max_flushed_lsn:{lsn:5311387238}, max_flushed_end_lsn:{lsn:5311387831}, max_flushed_log_pid:4, committed_end_lsn:{lsn:5311387238}, last_slide_log_id:8886934, last_slide_scn:{val:1747289697900328000, v:0}, last_slide_lsn:{lsn:5311386719}, last_slide_end_lsn:{lsn:5311387238}, last_slide_log_pid:4, last_slide_log_accum_checksum:3961629555, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f1a97bc4530}, [StateMgr]={this:0x7f1a97bc89b0, self:"127.0.0.1:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:-1, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7f1a97bcf470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"127.0.0.1:2882", alive_paxos_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:4427296652}, prev_mode_pid:4}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:4, config_seq:5}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f1a97bc5c70}, [ModeMgr]={palf_id:1, self:"127.0.0.1:2882", applied_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:5167067136}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, log_config_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:4427296652}, prev_mode_pid:4}, log_snapshot_meta:{version:2, base_lsn:{lsn:5167067136}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:118784}, readable_log_tail:{lsn:118784}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5368381440}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:131, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66985984, block_header_serialize_buf_:0x7f1a97bcabb8, flashback_version:0}, log_storage:{log_tail:{lsn:5311387831}, readable_log_tail:{lsn:5311387831}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:79, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:133, min_block_id:61, max_block_id:80, curr_writable_block_id:79}, logical_block_size_:67104768, curr_block_writable_size_:56993609, block_header_serialize_buf_:0x7f1a97bcc8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:-1, this:0x7f1a97bc90f0}, [Reconfirm]={palf_id:1, self:"127.0.0.1:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f1a97bc8b30}) [2025-05-15 09:40:31.931330] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=232][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:31.931354] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", ack_info_list=[]) [2025-05-15 09:40:31.931364] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:31.931382] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=16] current log_task status(palf_id=1, self="127.0.0.1:2882", log_id=8886935, log_task={header:{begin_lsn:{lsn:5311387238}, end_lsn:{lsn:5311387831}, log_id:8886935, min_scn:{val:1747289698016024000, v:0}, max_scn:{val:1747289698016024000, v:0}, data_len:537, proposal_id:4, prev_lsn:{lsn:5311386719}, prev_proposal_id:4, committed_end_lsn:{lsn:5311387238}, data_checksum:2749435567, accum_checksum:2227909955, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1747300521030351, freeze_ts:1747300521030351, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:-1747300521030352, submit_to_flush cost time:0}, ack_info_list=[]) [2025-05-15 09:40:31.931418] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:396) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4018] fail to get server region(ret=-4018, server="127.0.0.1:2882") [2025-05-15 09:40:31.931433] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4018] get_server_region failed(server="127.0.0.1:2882") [2025-05-15 09:40:31.931442] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:398) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:31.931458] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2766) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:31.932888] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4815586610, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:31.932922] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:31.932933] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1aaefec2b0) [2025-05-15 09:40:31.932950] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4815586610, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:31.937682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.947788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.956746] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=36] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:31.957897] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.958221] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:31.958252] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-05-15 09:40:31.958298] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=45] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:31.958321] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:31.958342] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11794848666, flush_io_finished_ret_:0, flush_io_finished_round_:1506, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:31.959325] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4531][T1002_Occam][T1002][YB427F000001-000635291EDC65C7-0-0] [lt=20] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.967530] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:38, tid:3961}]) [2025-05-15 09:40:31.968007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.978118] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.988231] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.992894] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552827, support_ssl=false) [2025-05-15 09:40:31.992965] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] sm conn init succ(conn.sessid_=3221552827, sess.client_addr_="127.0.0.1:45100") [2025-05-15 09:40:31.992981] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552827, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.993218] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:31.993257] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:31.993294] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:31.993324] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:31.993344] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:31.993353] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:31.993384] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.993417] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552827, proxy_sessid=0) [2025-05-15 09:40:31.993464] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=19][errcode=0] server close connection(sessid=3221552827, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:31.993481] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552827, proxy_sessid=0) [2025-05-15 09:40:31.993500] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=19][errcode=-4016] session is null [2025-05-15 09:40:31.993518] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905A-0-0] [lt=14] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552827, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=3908101, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:31.993576] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552827, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031992423, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:138, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:31.993600] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552827, proxy_sessid=0, ret=0) [2025-05-15 09:40:31.993612] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552827, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302031992423, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:138, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:31.993629] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552827) [2025-05-15 09:40:31.993647] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552827, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:31.995126] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=53] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:31.995180] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=29] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:31.995377] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD6-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.995399] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD6-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.995416] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD6-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.995437] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD6-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.995449] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD6-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.995467] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:31.995484] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.995516] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.995526] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302031995351, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:31.995545] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.995554] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:31.995630] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:31.996906] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD7-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.996926] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD7-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:31.996938] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD7-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:31.996956] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD7-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:31.996965] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CD7-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:31.996979] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:31.996988] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:31.997001] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:31.997009] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302031996897, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:31.997025] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:31.997034] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:31.997076] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.997261] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.997284] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:31.997293] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:31.997308] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:31.997317] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:31.997332] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:31.997351] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:31.997363] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:31.997402] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:31.997414] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:31.998341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:31.998379] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:32.001278] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.002114] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12334250743, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.002140] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.002169] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.002191] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.002200] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.002221] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=12334250743, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.002238] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.002256] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.002266] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.002280] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.002291] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.002306] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.002314] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032002094) [2025-05-15 09:40:32.006499] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-05-15 09:40:32.008460] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.012649] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=58] tx gc loop thread is running(MTL_ID()=1002) [2025-05-15 09:40:32.012679] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=28] try gc retain ctx [2025-05-15 09:40:32.012687] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-05-15 09:40:32.012718] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=13] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289693216794002, v:0}, keep_alive_scn_:{val:1747289693216794002, v:0}, update_ts_:1747302032012465}) [2025-05-15 09:40:32.013301] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=23] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:32.013390] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=77] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:0, v:0}, keep_alive_scn_:{val:0, v:0}, update_ts_:0}) [2025-05-15 09:40:32.013980] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:32.014202] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.018579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.023844] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12334295143, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.023877] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.023926] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=48][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.023947] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.023957] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.023968] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.023977] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.023990] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032023827) [2025-05-15 09:40:32.024075] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:32.024106] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28] get_min_unreplayed_log_info(lsn={lsn:5311387238}, scn={val:1747289697900328001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300529803179, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:5311387238}, next_to_submit_scn_:{val:1747289697900328001, v:0}, base_lsn_:{lsn:5167067136}, base_scn_:{val:1747274408663046000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a72405000, next_round_pread_size:3678208, curr_read_pos:2755105, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2755105, log_storage_:{start_lsn:{lsn:5308632133}, end_lsn:{lsn:5311387238}, read_buf:{buf_len_:3747840, buf_:0x7f1a72405000}, block_size:67104768, log_storage_:0x7f1a97bcbbf0, log_storage_:{log_tail:{lsn:5311387831}, readable_log_tail:{lsn:5311387831}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:79, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:133, min_block_id:61, max_block_id:80, curr_writable_block_id:79}, logical_block_size_:67104768, curr_block_writable_size_:56993609, block_header_serialize_buf_:0x7f1a97bcc8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697900328000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:431, scn_:{val:1747289697900328000, v:0}, data_checksum:4152786848, flag:25505}}, init_mode_version:0, accumulate_checksum:3961629555, curr_entry_is_padding:0, padding_entry_size:463, padding_entry_scn:{val:1747289697900328000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:76, read_io_size:145543168, read_disk_cost_ts:2159331, start_lsn:{lsn:5167067136}}}}}}) [2025-05-15 09:40:32.024204] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=96] get wrs ts(ls_id={id:1}, delta=12334123706, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.024217] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.024239] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.024259] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.024268] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.024288] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.024302] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.024310] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032024067) [2025-05-15 09:40:32.025446] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4606][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=25] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:32.028690] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.029553] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.029580] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12242149, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.029600] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12242149, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.029619] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.029631] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=12242149, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.029649] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.029671] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.029691] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.029710] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.029719] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.029738] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.029761] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032029758, replica_locations:[]}) [2025-05-15 09:40:32.029783] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.029813] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.029829] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:32.029851] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.029905] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11795920381, cache_obj->added_lc()=false, cache_obj->get_object_id()=15746, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.031077] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:32.031135] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10] LogLoopThread round_cost_time(us)(round_cost_time=402) [2025-05-15 09:40:32.031183] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.031210] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.031229] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.031248] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.031257] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.031275] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.031296] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.031309] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.031327] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.031336] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.031356] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.031376] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032031374, replica_locations:[]}) [2025-05-15 09:40:32.031447] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=12240234, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=224, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.038801] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.048914] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.054849] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.054897] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.054911] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC967A-0-0] [lt=32][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302032054880}) [2025-05-15 09:40:32.054930] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:32.054951] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:32.054974] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.054995] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.055349] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.055364] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:32.055376] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.055391] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.059020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.061276] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552831, support_ssl=false) [2025-05-15 09:40:32.061334] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552831, sess.client_addr_="172.22.0.1:45328") [2025-05-15 09:40:32.061349] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552831, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.061506] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.061534] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.061569] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.061606] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=36][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.061624] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.061633] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.061659] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.061679] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552831, proxy_sessid=0) [2025-05-15 09:40:32.061711] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=16][errcode=0] server close connection(sessid=3221552831, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.061724] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552831, proxy_sessid=0) [2025-05-15 09:40:32.061732] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.061747] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9050-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552831, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.061802] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552831, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032060988, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.061823] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552831, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.061833] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552831, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032060988, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.061848] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552831) [2025-05-15 09:40:32.061862] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552831, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.065642] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4577][T1002_TimerWK0_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=21] ====== check clog disk timer task ====== [2025-05-15 09:40:32.065667] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4577][T1002_TimerWK0_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=21] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:32.067754] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:41, tid:4542}]) [2025-05-15 09:40:32.068581] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:32.068613] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.068630] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:32.068639] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:32.068654] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:32.068741] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=87][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:32.069130] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.079245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.089359] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.099471] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.100373] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552832, support_ssl=false) [2025-05-15 09:40:32.100415] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552832, sess.client_addr_="172.22.0.1:45330") [2025-05-15 09:40:32.100427] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de04030, session_id:3221552832, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.100714] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.100744] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.100771] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552833, support_ssl=false) [2025-05-15 09:40:32.100783] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.100815] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=31][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.100835] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.100843] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.100855] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221552833, sess.client_addr_="172.22.0.1:45344") [2025-05-15 09:40:32.100875] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.100874] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221552833, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.100900] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552832, proxy_sessid=0) [2025-05-15 09:40:32.100909] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552834, support_ssl=false) [2025-05-15 09:40:32.100942] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=16][errcode=0] server close connection(sessid=3221552832, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.100944] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221552834, sess.client_addr_="172.22.0.1:45348") [2025-05-15 09:40:32.100958] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552832, proxy_sessid=0) [2025-05-15 09:40:32.100967] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.100958] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552834, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.100982] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552835, support_ssl=false) [2025-05-15 09:40:32.100978] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908B-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552832, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.101014] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552835, sess.client_addr_="172.22.0.1:45354") [2025-05-15 09:40:32.101030] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552835, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.101033] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221552832, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032100087, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.101056] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552832, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.101065] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221552832, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032100087, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.101080] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552832) [2025-05-15 09:40:32.101095] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552832, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.101135] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.101180] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.101195] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.101204] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.101216] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.101225] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.101224] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.101234] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.101243] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.101263] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.101262] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=19] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.101255] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.101272] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.101278] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552833, proxy_sessid=0) [2025-05-15 09:40:32.101279] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.101285] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.101289] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.101293] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.101301] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.101300] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=8][errcode=0] server close connection(sessid=3221552833, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.101298] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.101314] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552833, proxy_sessid=0) [2025-05-15 09:40:32.101319] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.101322] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.101329] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552835, proxy_sessid=0) [2025-05-15 09:40:32.101325] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=17] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.101329] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905B-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552833, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.101344] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552834, proxy_sessid=0) [2025-05-15 09:40:32.101343] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.101348] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=9][errcode=0] server close connection(sessid=3221552835, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.101360] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552835, proxy_sessid=0) [2025-05-15 09:40:32.101368] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.101373] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=11][errcode=0] server close connection(sessid=3221552834, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.101374] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221552833, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032100087, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.101376] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908C-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552835, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.101389] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552834, proxy_sessid=0) [2025-05-15 09:40:32.101397] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552833, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.101399] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:32.101409] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221552833, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032100087, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.101410] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9051-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552834, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.101436] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] mark session id unused(conn.sessid_=3221552833) [2025-05-15 09:40:32.101456] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552833, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.101511] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552835, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302032101147, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.101543] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552834, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302032101147, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.101568] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552835, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.101577] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] kill and revert session(conn.sessid_=3221552834, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.101618] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552835, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302032101147, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.101644] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] mark session id unused(conn.sessid_=3221552835) [2025-05-15 09:40:32.101656] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552835, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.101680] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552834, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302032101147, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.101699] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552834) [2025-05-15 09:40:32.101709] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552834, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.101812] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221552836, support_ssl=false) [2025-05-15 09:40:32.101856] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552836, sess.client_addr_="172.22.0.1:45356") [2025-05-15 09:40:32.101874] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221552836, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.101899] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552837, support_ssl=false) [2025-05-15 09:40:32.101956] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552837, sess.client_addr_="172.22.0.1:45372") [2025-05-15 09:40:32.101978] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221552837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.102016] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552838, support_ssl=false) [2025-05-15 09:40:32.102021] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.102046] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552838, sess.client_addr_="172.22.0.1:45384") [2025-05-15 09:40:32.102054] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.102065] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.102064] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552838, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.102078] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.102088] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.102094] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.102106] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.102123] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552836, proxy_sessid=0) [2025-05-15 09:40:32.102138] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] ====== tenant freeze timer task ====== [2025-05-15 09:40:32.102142] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=8][errcode=0] server close connection(sessid=3221552836, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.102155] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552836, proxy_sessid=0) [2025-05-15 09:40:32.102162] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.102169] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905C-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552836, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.102166] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.102202] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032102158) [2025-05-15 09:40:32.102222] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302031902102, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.102226] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.102223] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221552836, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032101147, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.102247] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552836, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.102255] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12334351009, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.102263] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221552836, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032101147, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.102273] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.102280] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552836) [2025-05-15 09:40:32.102275] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.102286] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.102295] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.102292] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552836, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.102303] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.102299] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.102315] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.102318] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.102318] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.102323] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.102329] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.102335] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.102340] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.102345] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12334351009, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.102353] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.102358] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552838, proxy_sessid=0) [2025-05-15 09:40:32.102362] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.102359] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.102369] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.102377] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=9][errcode=0] server close connection(sessid=3221552838, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.102376] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.102381] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.102391] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.102392] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552838, proxy_sessid=0) [2025-05-15 09:40:32.102396] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552837, proxy_sessid=0) [2025-05-15 09:40:32.102399] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.102399] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.102409] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.102405] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908D-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552838, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.102414] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=8][errcode=0] server close connection(sessid=3221552837, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.102417] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.102426] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552837, proxy_sessid=0) [2025-05-15 09:40:32.102429] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032102235) [2025-05-15 09:40:32.102434] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:32.102445] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9052-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552837, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.102451] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552838, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302032102203, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.102498] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221552837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032102203, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.102527] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221552838, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.102538] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] kill and revert session(conn.sessid_=3221552837, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.102552] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552838, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302032102203, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.102574] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552838) [2025-05-15 09:40:32.102591] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552838, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.102622] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221552837, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032102203, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.102641] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552837) [2025-05-15 09:40:32.102665] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] connection close(sessid=3221552837, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.103303] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552839, support_ssl=false) [2025-05-15 09:40:32.103342] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552839, sess.client_addr_="172.22.0.1:45392") [2025-05-15 09:40:32.103358] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552839, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.103522] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.103552] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.103565] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.103576] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.103591] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.103599] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.103618] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.103635] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552839, proxy_sessid=0) [2025-05-15 09:40:32.103653] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=7][errcode=0] server close connection(sessid=3221552839, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.103676] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=22][errcode=-4018] get session fail(ret=-4018, sessid=3221552839, proxy_sessid=0) [2025-05-15 09:40:32.103683] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.103697] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905D-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552839, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.103745] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552839, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032103262, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.103768] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552839, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.103776] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552839, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032103262, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.103795] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552839) [2025-05-15 09:40:32.103807] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552839, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.104815] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552840, support_ssl=false) [2025-05-15 09:40:32.104865] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552840, sess.client_addr_="172.22.0.1:45396") [2025-05-15 09:40:32.104880] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552840, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.105080] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.105114] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.105132] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.105130] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552841, support_ssl=false) [2025-05-15 09:40:32.105152] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.105160] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.105167] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.105188] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.105192] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221552841, sess.client_addr_="172.22.0.1:45398") [2025-05-15 09:40:32.105205] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552840, proxy_sessid=0) [2025-05-15 09:40:32.105223] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=8][errcode=0] server close connection(sessid=3221552840, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.105212] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552841, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.105237] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552840, proxy_sessid=0) [2025-05-15 09:40:32.105245] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.105253] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908E-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552840, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.105304] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552840, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032104325, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.105327] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552840, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.105328] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552842, support_ssl=false) [2025-05-15 09:40:32.105337] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552840, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302032104325, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.105353] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552840) [2025-05-15 09:40:32.105366] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552840, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.105376] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552842, sess.client_addr_="172.22.0.1:45402") [2025-05-15 09:40:32.105377] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.105393] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552842, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.105407] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.105429] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.105452] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.105461] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.105475] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.105504] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.105520] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552841, proxy_sessid=0) [2025-05-15 09:40:32.105540] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=8][errcode=0] server close connection(sessid=3221552841, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.105554] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552841, proxy_sessid=0) [2025-05-15 09:40:32.105561] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:32.105562] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.105576] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9053-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552841, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.105594] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.105607] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.105619] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.105634] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.105631] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552841, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032104325, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.105641] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.105660] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.105657] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552841, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.105669] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=7][errcode=-4018] get session fail(ret=-4018, sessid=3221552842, proxy_sessid=0) [2025-05-15 09:40:32.105672] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552841, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032104325, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.105690] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552841) [2025-05-15 09:40:32.105691] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=11][errcode=0] server close connection(sessid=3221552842, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.105702] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552842, proxy_sessid=0) [2025-05-15 09:40:32.105709] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.105709] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552841, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.105715] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905E-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552842, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.105761] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552842, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032105383, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.105788] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552842, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.105798] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552842, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032105383, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.105818] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552842) [2025-05-15 09:40:32.105830] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552842, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.109601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.111621] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:32.114267] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.114291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.114314] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.114335] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.114385] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=49][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.114401] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.114412] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.114427] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.114437] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302032114239, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.114455] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.114463] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.114553] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.114570] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.114578] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.114592] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.114740] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:32.114760] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:32.114769] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=9] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:32.114791] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=21] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:32.119757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.119931] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:32.123893] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.123926] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032123885) [2025-05-15 09:40:32.123949] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302031923927, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.123989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12334395514, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.124013] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.124065] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=50][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.124088] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.124098] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.124119] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.124127] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.124132] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.124142] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032123968) [2025-05-15 09:40:32.124152] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032124128) [2025-05-15 09:40:32.124171] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302031924077, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.124203] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12334223021, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.124243] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=37] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.124265] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.124283] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.124292] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.124309] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.124318] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.124332] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032124190) [2025-05-15 09:40:32.127585] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=47] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:32.127617] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-05-15 09:40:32.127630] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:32.127652] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:32.127673] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11795018007, flush_io_finished_ret_:0, flush_io_finished_round_:1506, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:32.129879] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.131608] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.131636] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12140100, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.131657] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12140100, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.131676] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.131688] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=12140100, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.131707] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.131729] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.131751] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.131772] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.131782] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.131802] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.131826] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032131823, replica_locations:[]}) [2025-05-15 09:40:32.131858] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=30] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.131890] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.131906] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:32.131950] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.132009] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11796022488, cache_obj->added_lc()=false, cache_obj->get_object_id()=15747, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.133402] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.133428] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.133447] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.133460] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.133475] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.133506] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.133519] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.133537] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.133554] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.133563] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.133582] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.133600] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032133600, replica_locations:[]}) [2025-05-15 09:40:32.133676] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12138006, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=225, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.139997] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.150113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.160232] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.170345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.173456] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.173841] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.174431] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.174750] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.179682] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=24] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:32.179715] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:32.179729] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:32.179749] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=19][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:32.179769] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=18][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:32.179783] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:32.179796] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:32.179811] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:32.179822] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:32.179830] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=7][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:32.179844] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=14][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:32.179853] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:32.179875] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=20][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:32.179886] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:32.179912] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=15][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:32.179927] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.179948] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=18][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.179968] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=17][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:32.179985] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=15][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:32.180004] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=16][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:32.180013] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:32.180039] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:32.180065] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=21][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:32.180079] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:32.180086] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:32.180104] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:32.180123] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646D-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.180132] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:32.180149] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.180158] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:32.180173] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=14][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:32.180181] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302032179421, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.180197] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:32.180206] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.180297] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:32.180315] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=17][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:32.180325] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=9][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:32.180340] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=15][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:32.180354] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:32.180370] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=15][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:32.180378] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646D-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:32.180457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.190583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.191152] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:32.200695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.201403] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.202230] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:32.202268] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31] get_min_unreplayed_log_info(lsn={lsn:934287681}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302032108621, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:934287681}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:872361984}, base_scn_:{val:1747274406063927000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69605000, next_round_pread_size:3678208, curr_read_pos:3075493, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3075493, log_storage_:{start_lsn:{lsn:931212188}, end_lsn:{lsn:934287681}, read_buf:{buf_len_:3747840, buf_:0x7f1a69605000}, block_size:67104768, log_storage_:0x7f1a73fd1bf0, log_storage_:{log_tail:{lsn:934288498}, readable_log_tail:{lsn:934288498}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:13, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:143, min_block_id:0, max_block_id:14, curr_writable_block_id:13}, logical_block_size_:67104768, curr_block_writable_size_:5178254, block_header_serialize_buf_:0x7f1a73fd28b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:2923900223, flag:16486}}, init_mode_version:0, accumulate_checksum:910368143, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}}}}) [2025-05-15 09:40:32.202365] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=94] get wrs ts(ls_id={id:1}, delta=12334450285, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.202383] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.202413] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.202433] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.202444] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.202464] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12334450285, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.202483] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.202513] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.202523] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.202536] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.202548] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.202562] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.202570] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032202216) [2025-05-15 09:40:32.210806] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.214193] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.214217] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.214234] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.214257] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.214269] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.214285] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.214294] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.214309] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.214319] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302032214177, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.214336] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.214344] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.214401] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.214416] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.214425] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.214440] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.221280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.223966] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12334495839, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.223997] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.224046] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=46][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.224068] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.224077] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.224095] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.224111] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.224118] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032223944) [2025-05-15 09:40:32.224201] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=12334323346, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.224225] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.224249] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.224267] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.224276] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.224294] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.224308] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.224315] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032224188) [2025-05-15 09:40:32.231176] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:32.231285] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302032231096, register_time_us=-1, curr_time_us=1747302032231282) [2025-05-15 09:40:32.231314] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:32.231950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.233809] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:32.233834] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.233846] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12037892, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.233928] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=82][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=12037892, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.233939] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.233955] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=12037892, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.233974] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.233995] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.234007] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.234026] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.234035] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.234054] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.234075] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032234074, replica_locations:[]}) [2025-05-15 09:40:32.234099] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.234127] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.234142] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:32.234171] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.234222] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11796124699, cache_obj->added_lc()=false, cache_obj->get_object_id()=15748, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.235357] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.235381] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.235395] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.235411] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.235420] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.235438] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.235451] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.235468] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.235485] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.235514] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.235526] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.235545] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032235544, replica_locations:[]}) [2025-05-15 09:40:32.235611] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=12036071, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=226, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.237752] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=27][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.237783] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.237809] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.237829] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=19][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.237840] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.237860] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=19][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.237889] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=25][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.237912] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.237934] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=22][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.237945] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.237965] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.237998] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=15] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.238016] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=15] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:32.239592] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:32.239633] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=38][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:32.239656] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=22][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:32.239676] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=19][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:32.239688] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:32.239703] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:32.239717] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:32.239731] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:32.239740] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:32.239754] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:32.239762] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:32.239777] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:32.239786] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:32.239801] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:32.239817] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:32.239832] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.239843] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.239859] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:32.239869] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:32.239885] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:32.239892] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:32.239914] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:32.239939] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=22][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:32.239946] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:32.239959] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:32.239973] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:32.239991] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.240000] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:32.240016] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=15][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.240025] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:32.240033] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:32.240048] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302032239374, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.240060] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:32.240074] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.240138] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=9][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:32.240155] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=15][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:32.240164] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=9][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:32.240179] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=14][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:32.240190] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:32.240206] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=15][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:32.240214] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BC-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:32.242068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.252182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.253473] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.253507] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:32.253518] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:32.253538] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:32.253551] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:32.253596] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:32.258152] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=119] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:32.258184] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=25] temporary file module use no blocks [2025-05-15 09:40:32.258201] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:32.258221] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:32.258242] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11795148570, flush_io_finished_ret_:0, flush_io_finished_round_:1506, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:32.262293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.263875] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65C9-0-0] [lt=15] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.265304] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=30] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:32.272407] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.274076] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4501][T1001_ArcSender][T1001][YB427F000001-000635291CBC6112-0-0] [lt=53][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:32.282526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.292639] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.296775] WDIAG load_file_to_string (utility.h:673) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:32.296798] WDIAG get_ethernet_speed (utility.cpp:575) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:32.296808] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:32.296826] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:32.301472] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.302281] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001) [2025-05-15 09:40:32.302306] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.302334] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032302274) [2025-05-15 09:40:32.302354] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302032102234, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.302386] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12334550726, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.302412] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.302441] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.302463] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.302480] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.302502] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1001}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:32.302529] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] get_min_unreplayed_log_info(lsn={lsn:747348431}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302032208897, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:747348431}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:738152448}, base_scn_:{val:1747289697751194001, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a5a205000, next_round_pread_size:3678208, curr_read_pos:1839725, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1839725, log_storage_:{start_lsn:{lsn:745508706}, end_lsn:{lsn:747348431}, read_buf:{buf_len_:3747840, buf_:0x7f1a5a205000}, block_size:67104768, log_storage_:0x7f1a73f61bf0, log_storage_:{log_tail:{lsn:747348553}, readable_log_tail:{lsn:747348553}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:11, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:140, min_block_id:0, max_block_id:12, curr_writable_block_id:11}, logical_block_size_:67104768, curr_block_writable_size_:57908663, block_header_serialize_buf_:0x7f1a73f628b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:154833137, flag:60041}}, init_mode_version:0, accumulate_checksum:2356512705, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}}}}) [2025-05-15 09:40:32.302620] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=91] get wrs ts(ls_id={id:1001}, delta=12334550726, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.302643] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.302663] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.302674] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.302688] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.302700] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.302714] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.302722] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032302365) [2025-05-15 09:40:32.302749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.311882] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:32.312882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.314779] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.314809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=28][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.314844] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.314872] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.314893] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.314914] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.314935] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.314955] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.314975] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302032314749, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.314998] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.315010] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.315102] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.315121] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.315134] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.315154] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.317833] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=49] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5900, clean_start_pos=12200000, clean_num=200000) [2025-05-15 09:40:32.322992] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.324008] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.324043] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032324001) [2025-05-15 09:40:32.324066] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302032123967, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.324108] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12334596254, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.324138] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.324179] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.324207] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.324218] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.324237] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.324252] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.324251] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.324262] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032324085) [2025-05-15 09:40:32.324278] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032324247) [2025-05-15 09:40:32.324290] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302032124190, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.324324] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12334423761, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.324359] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.324383] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.324402] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.324412] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.324431] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.324445] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.324452] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032324312) [2025-05-15 09:40:32.333104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.335764] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.335796] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11935943, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.335863] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=67][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11935943, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.335874] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.335891] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=11935943, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.335911] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.335932] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.335955] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.335974] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.335984] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.336003] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.336025] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032336023, replica_locations:[]}) [2025-05-15 09:40:32.336047] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.336078] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.336094] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:32.336136] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.336200] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11796226674, cache_obj->added_lc()=false, cache_obj->get_object_id()=15749, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.337536] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.337560] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.337585] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.337604] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.337612] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.337629] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.337649] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.337661] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.337678] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.337687] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.337706] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.337726] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032337725, replica_locations:[]}) [2025-05-15 09:40:32.337799] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11933883, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=227, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.343217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.353144] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:32.353177] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.353194] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:32.353203] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:32.353218] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:32.353300] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:32.353310] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.353325] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:32.353326] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.353334] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:32.353344] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:32.353397] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:32.357998] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=17] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:32.358072] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:32.358142] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:32.358208] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:32.358242] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:8482, total:8498) [2025-05-15 09:40:32.358284] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:32.358336] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=47] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:8482, total:8498) [2025-05-15 09:40:32.358397] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=41] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:32.358422] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:32.358441] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:32.358462] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] successfully alloc block(macro_id={[ver=1,mode=0,seq=628767][2nd=1894]}) [2025-05-15 09:40:32.363433] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.373544] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.381605] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4292][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=1] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}) [2025-05-15 09:40:32.383654] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.389056] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4776][BlackListServic][T1][YB427F000001-000635291F1C6308-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:32.389088] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4776][BlackListServic][T1][YB427F000001-000635291F1C6308-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2025-05-15 09:40:32.389101] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4776][BlackListServic][T1][YB427F000001-000635291F1C6308-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:32.389121] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4776][BlackListServic][T1][YB427F000001-000635291F1C6308-0-0] [lt=19][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2025-05-15 09:40:32.389133] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4776][BlackListServic][T1][YB427F000001-000635291F1C6308-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:32.389149] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4776][BlackListServic][T1][YB427F000001-000635291F1C6308-0-0] [lt=15][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.389221] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4776][BlackListServic][T1][YB427F000001-000635291F1C6308-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.389252] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=0] start to print blacklist info [2025-05-15 09:40:32.389335] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=15] ls blacklist refresh finish(cost_time=2055) [2025-05-15 09:40:32.393768] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.395969] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.395969] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.396107] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=29] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.396300] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.396342] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.396441] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.396715] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:32.401520] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.402370] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12334651137, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.402399] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.402428] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.402450] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.402463] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.402483] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=12334651137, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.402512] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.402531] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.402544] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.402555] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.402572] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.402588] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.402596] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032402349) [2025-05-15 09:40:32.404869] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.412665] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.414365] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.414388] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.414406] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.414426] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.414442] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.414452] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.414468] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.414479] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.414503] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302032414342, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.414514] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.414523] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.414592] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.414607] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.414625] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.414633] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.414982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.424073] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12334695637, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.424102] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.424200] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=97][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.424228] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.424244] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.424257] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.424272] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.424279] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032424056) [2025-05-15 09:40:32.424318] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12334523144, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.424341] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.424364] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.424383] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.424392] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.424410] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.424425] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.424433] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032424307) [2025-05-15 09:40:32.425093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.431270] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=26] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-05-15 09:40:32.435210] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.437930] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.437958] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11833770, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.437978] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11833770, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.437990] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.437999] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=11833770, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.438017] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.438039] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.438059] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.438078] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.438087] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.438106] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.438127] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032438125, replica_locations:[]}) [2025-05-15 09:40:32.438149] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.438175] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.438190] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:32.438212] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.438256] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11796328733, cache_obj->added_lc()=false, cache_obj->get_object_id()=15750, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.439409] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.439432] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.439451] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.439470] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.439478] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.439507] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.439519] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.439537] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.439555] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.439564] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.439583] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.439600] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032439599, replica_locations:[]}) [2025-05-15 09:40:32.439671] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11832011, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=228, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.445322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.455431] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.465542] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.468576] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:31, tid:4482}]) [2025-05-15 09:40:32.475658] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.485776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.495890] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.497459] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.497506] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=48][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.497517] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:32.497533] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:32.497543] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:32.497559] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:32.497582] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:32.497596] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:32.497638] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=18] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:32.497651] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.499171] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.499199] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=27][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.499210] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:32.499226] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=15][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:32.499235] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=8][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:32.499259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=14][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:32.499281] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=20][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.499291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_TntSharedTimer][T1002][YB427F000001-000635291F7C65FF-0-0] [lt=9][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5157, ret=-5157) [2025-05-15 09:40:32.499306] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:32.499328] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:32.499356] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:32.499382] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78ca060, start=1747302032499141, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:32.499399] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:32.499547] INFO [STORAGE] runTimerTask (ob_tenant_restore_info_mgr.cpp:281) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=33] refresh restore info [2025-05-15 09:40:32.501587] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.502409] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.502441] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032502401) [2025-05-15 09:40:32.502462] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302032302365, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.502516] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12334750493, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.502543] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.502573] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.502596] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.502606] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.502627] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=12334750493, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.502645] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.502663] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.502682] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.502689] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.502706] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.502721] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.502729] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032502481) [2025-05-15 09:40:32.502842] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:32.502883] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=37] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:32.505999] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.514456] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.514479] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.514504] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.514525] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.514534] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.514556] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.514567] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.514582] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.514591] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302032514439, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.514607] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.514616] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.514675] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.514690] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.514697] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.514712] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.516113] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.518091] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:32.523073] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4289][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=26] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:32.524120] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.524160] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032524114) [2025-05-15 09:40:32.524181] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302032324084, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.524212] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12334796096, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.524233] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.524262] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.524282] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.524291] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.524305] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.524321] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.524328] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032524197) [2025-05-15 09:40:32.524370] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.524402] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032524366) [2025-05-15 09:40:32.524421] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302032324307, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.524443] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12334623603, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.524468] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.524500] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.524512] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.524527] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.524539] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.524552] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.524559] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032524433) [2025-05-15 09:40:32.526216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.536327] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.538939] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=23] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:32.539830] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.539858] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11731879, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.539880] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11731879, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.539899] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.539910] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=11731879, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.539953] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=42][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.539976] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.539998] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.540017] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.540026] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.540046] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.540071] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032540068, replica_locations:[]}) [2025-05-15 09:40:32.540095] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.540129] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.540145] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:32.540189] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.540256] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11796430729, cache_obj->added_lc()=false, cache_obj->get_object_id()=15751, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.541672] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.541696] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.541716] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.541736] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.541745] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.541747] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:32.541762] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.541783] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.541801] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.541818] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.541827] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.541846] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.541864] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032541864, replica_locations:[]}) [2025-05-15 09:40:32.541942] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11729739, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=229, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.546440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.555835] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.555863] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=27] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.555876] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:32.555903] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:32.555921] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:32.555941] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.556299] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.556320] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:32.556333] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:32.556561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.562940] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552851, support_ssl=false) [2025-05-15 09:40:32.563014] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221552851, sess.client_addr_="172.22.0.1:45404") [2025-05-15 09:40:32.563034] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552851, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.563258] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.563303] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.563336] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.563365] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.563385] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.563395] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.563446] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=41] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.563469] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552851, proxy_sessid=0) [2025-05-15 09:40:32.563523] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=16][errcode=0] server close connection(sessid=3221552851, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.563540] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552851, proxy_sessid=0) [2025-05-15 09:40:32.563548] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:32.563565] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC908F-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552851, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.563629] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552851, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032563052, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.563657] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552851, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.563673] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552851, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032563052, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.563696] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552851) [2025-05-15 09:40:32.563713] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552851, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.565265] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6692-0-0] [lt=21][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.565289] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6692-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.565310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6692-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.565332] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6692-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.565348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.565359] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.565376] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.565385] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.565401] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302032565240, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.565420] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=19][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.565429] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:32.565516] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=17][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:32.565535] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=17][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:32.565544] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=9][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:32.565553] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=8][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:32.565570] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=15][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:32.565581] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=9][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:32.565595] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6692-0-0] [lt=14][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:32.568806] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:424, tid:4579}]) [2025-05-15 09:40:32.568864] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.578985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.589101] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.599215] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.601645] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.602179] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552853, support_ssl=false) [2025-05-15 09:40:32.602316] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221552853, sess.client_addr_="172.22.0.1:45412") [2025-05-15 09:40:32.602338] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552853, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.602508] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12334850969, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.602549] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=31] new mysql sessid created(conn.sessid_=3221552854, support_ssl=false) [2025-05-15 09:40:32.602554] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.602548] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.602578] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.602595] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.602603] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552854, sess.client_addr_="172.22.0.1:45422") [2025-05-15 09:40:32.602620] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.602627] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.602657] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=29][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.602668] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.602665] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.602676] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.602690] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.602700] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.602721] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552853, proxy_sessid=0) [2025-05-15 09:40:32.602721] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1001}, delta=12334850969, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.602756] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=16][errcode=0] server close connection(sessid=3221552853, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.602747] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.602771] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552853, proxy_sessid=0) [2025-05-15 09:40:32.602780] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:32.602775] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.602789] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9054-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552853, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.602795] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221552855, support_ssl=false) [2025-05-15 09:40:32.602801] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.602819] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.602817] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552856, support_ssl=false) [2025-05-15 09:40:32.602851] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.602881] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.602891] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552853, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.602903] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552855, sess.client_addr_="172.22.0.1:45436") [2025-05-15 09:40:32.602901] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221552856, sess.client_addr_="172.22.0.1:45426") [2025-05-15 09:40:32.602903] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032602465) [2025-05-15 09:40:32.602917] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552853, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.602922] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552855, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.602931] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552853, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.602923] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552856, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.602952] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552853) [2025-05-15 09:40:32.602970] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.602969] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552853, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.603005] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.603030] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.603054] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.603066] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.603081] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.603100] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603108] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221552857, support_ssl=false) [2025-05-15 09:40:32.603118] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552854, proxy_sessid=0) [2025-05-15 09:40:32.603135] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221552857, sess.client_addr_="172.22.0.1:45446") [2025-05-15 09:40:32.603146] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552857, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.603153] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=18][errcode=0] server close connection(sessid=3221552854, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603168] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552854, proxy_sessid=0) [2025-05-15 09:40:32.603176] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:32.603180] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.603190] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC905F-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552854, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.603200] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.603201] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.603220] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=29][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.603242] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.603239] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.603252] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.603259] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.603262] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552854, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.603268] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.603268] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.603278] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552854, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.603284] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603298] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552854) [2025-05-15 09:40:32.603299] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552855, proxy_sessid=0) [2025-05-15 09:40:32.603297] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=29][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.603318] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=7][errcode=0] server close connection(sessid=3221552855, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603316] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.603314] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552854, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.603336] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552855, proxy_sessid=0) [2025-05-15 09:40:32.603335] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=18][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.603344] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:32.603355] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.603353] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603353] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9090-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552855, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.603373] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552856, proxy_sessid=0) [2025-05-15 09:40:32.603380] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.603385] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.603396] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.603404] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.603411] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.603411] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=22][errcode=0] server close connection(sessid=3221552856, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603428] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603419] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552855, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.603438] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552857, proxy_sessid=0) [2025-05-15 09:40:32.603430] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552856, proxy_sessid=0) [2025-05-15 09:40:32.603449] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=18][errcode=-4016] session is null [2025-05-15 09:40:32.603456] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=11][errcode=0] server close connection(sessid=3221552857, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.603456] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221552855, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.603469] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552857, proxy_sessid=0) [2025-05-15 09:40:32.603477] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.603468] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9055-0-0] [lt=17] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552856, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.603484] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9091-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552857, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.603522] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552855, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.603545] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552856, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.603563] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=40] mark session id unused(conn.sessid_=3221552855) [2025-05-15 09:40:32.603571] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552856, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.603580] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552856, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032602163, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.603595] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552856) [2025-05-15 09:40:32.603582] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552855, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.603614] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552856, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.603622] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552857, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032603232, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.603653] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221552857, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.603659] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552858, support_ssl=false) [2025-05-15 09:40:32.603695] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221552858, sess.client_addr_="172.22.0.1:45448") [2025-05-15 09:40:32.603705] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552857, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032603232, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.603712] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552858, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.603723] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552857) [2025-05-15 09:40:32.603736] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552857, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.603742] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552859, support_ssl=false) [2025-05-15 09:40:32.603823] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=43] sm conn init succ(conn.sessid_=3221552859, sess.client_addr_="172.22.0.1:45450") [2025-05-15 09:40:32.603851] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552859, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.604009] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.604064] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.604078] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.604094] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.604106] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.604113] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.604113] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.604131] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.604141] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552858, proxy_sessid=0) [2025-05-15 09:40:32.604159] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=8][errcode=0] server close connection(sessid=3221552858, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.604162] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.604170] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552858, proxy_sessid=0) [2025-05-15 09:40:32.604178] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:32.604185] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9060-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552858, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.604199] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=30][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.604228] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.604250] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.604237] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552858, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032603232, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.604259] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.604263] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552858, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.604275] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552858, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032603232, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.604293] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552858) [2025-05-15 09:40:32.604289] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.604306] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552858, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.604311] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552859, proxy_sessid=0) [2025-05-15 09:40:32.604352] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=18][errcode=0] server close connection(sessid=3221552859, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.604366] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552859, proxy_sessid=0) [2025-05-15 09:40:32.604375] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.604386] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9092-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552859, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.604439] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552859, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302032603232, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.604465] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552859, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.604473] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552859, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302032603232, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.604505] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] mark session id unused(conn.sessid_=3221552859) [2025-05-15 09:40:32.604521] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552859, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.604592] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552860, support_ssl=false) [2025-05-15 09:40:32.604633] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552860, sess.client_addr_="172.22.0.1:45466") [2025-05-15 09:40:32.604649] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552860, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.604792] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.604821] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.604840] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=47][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.604862] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.604871] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.604884] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.604899] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.604914] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552860, proxy_sessid=0) [2025-05-15 09:40:32.604934] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=8][errcode=0] server close connection(sessid=3221552860, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.604948] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552860, proxy_sessid=0) [2025-05-15 09:40:32.604956] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.604963] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9056-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552860, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.605006] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552860, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032604304, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.605036] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221552860, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.605045] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552860, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032604304, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.605063] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552860) [2025-05-15 09:40:32.605075] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552860, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.606096] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552861, support_ssl=false) [2025-05-15 09:40:32.606129] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552861, sess.client_addr_="172.22.0.1:45472") [2025-05-15 09:40:32.606146] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.606333] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.606360] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.606375] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.606392] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.606403] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.606413] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.606429] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.606442] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552861, proxy_sessid=0) [2025-05-15 09:40:32.606446] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552862, support_ssl=false) [2025-05-15 09:40:32.606466] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=13][errcode=0] server close connection(sessid=3221552861, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.606477] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552861, proxy_sessid=0) [2025-05-15 09:40:32.606484] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.606506] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552862, sess.client_addr_="172.22.0.1:45488") [2025-05-15 09:40:32.606500] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9061-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552861, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.606526] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552862, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.606556] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032605361, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.606580] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552861, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.606596] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552861, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302032605361, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.606616] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552861) [2025-05-15 09:40:32.606628] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552861, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.606691] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.606725] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.606747] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.606771] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.606780] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.606794] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.606791] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552863, support_ssl=false) [2025-05-15 09:40:32.606810] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.606828] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552862, proxy_sessid=0) [2025-05-15 09:40:32.606835] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552863, sess.client_addr_="172.22.0.1:45490") [2025-05-15 09:40:32.606850] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=9][errcode=0] server close connection(sessid=3221552862, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.606850] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552863, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.606865] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552862, proxy_sessid=0) [2025-05-15 09:40:32.606873] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:32.606881] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9093-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552862, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.606936] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552862, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032606421, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.606960] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552862, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.606970] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552862, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302032606421, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.606986] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552862) [2025-05-15 09:40:32.606995] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:32.607001] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552862, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.607028] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:32.607066] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:32.607095] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:32.607107] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:32.607129] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=21][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:32.607148] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.607172] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=21][errcode=-4018] get session fail(ret=-4018, sessid=3221552863, proxy_sessid=0) [2025-05-15 09:40:32.607212] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=23][errcode=0] server close connection(sessid=3221552863, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.607232] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552863, proxy_sessid=0) [2025-05-15 09:40:32.607253] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=20][errcode=-4016] session is null [2025-05-15 09:40:32.607283] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9057-0-0] [lt=28] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552863, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:32.607354] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552863, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032606421, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:32.607378] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552863, proxy_sessid=0, ret=0) [2025-05-15 09:40:32.607387] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552863, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302032606421, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:32.607406] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552863) [2025-05-15 09:40:32.607420] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552863, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:32.609332] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.614556] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.614578] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.614604] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.614622] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.614636] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.614648] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.614661] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.614672] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.614684] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302032614529, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.614698] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.614707] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.614777] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.614789] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.614799] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.614809] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.619466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.619600] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.619625] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.619644] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.620778] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:32.620807] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.620822] WDIAG [CLOG] nonblock_get_leader (ob_location_adapter.cpp:127) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] location adapter nonblock_get_leader failed(ret=-4721, id=1, leader="0.0.0.0:0") [2025-05-15 09:40:32.620835] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.620855] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:32.620865] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:32.620876] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:32.624195] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12334895469, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.624221] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.624247] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.624266] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.624277] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.624293] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.624305] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.624315] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032624175) [2025-05-15 09:40:32.624439] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12334724032, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.624459] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.624478] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.624503] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.624512] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.624523] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.624534] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.624544] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032624426) [2025-05-15 09:40:32.629592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.631363] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-05-15 09:40:32.631512] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:32.631531] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.631545] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.639704] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.642112] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.642140] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11629599, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.642158] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11629599, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.642172] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.642185] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=11629599, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.642200] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.642217] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.642235] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.642251] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.642261] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.642277] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.642297] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032642294, replica_locations:[]}) [2025-05-15 09:40:32.642315] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.642335] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:229, local_retry_times:229, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:32.642358] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.642369] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:32.642385] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.642396] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.642404] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:32.642424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:32.642439] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.642519] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11796532992, cache_obj->added_lc()=false, cache_obj->get_object_id()=15752, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:32.643963] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.643993] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.644030] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.644045] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.644060] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.644073] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:32.644083] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:32.644097] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:32.644111] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:32.644124] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.644137] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.644147] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.644161] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.644173] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032644173, replica_locations:[]}) [2025-05-15 09:40:32.644190] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.644204] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.644216] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.644231] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:32.644242] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:32.644252] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:32.644270] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:32.644285] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:32.644292] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:32.644303] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:32.644313] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:32.644322] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:32.644334] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:32.644345] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:32.644354] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:32.644363] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:32.644373] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:32.644382] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:32.644392] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:32.644407] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:32.644418] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.644428] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:32.644438] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:32.644453] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:32.644462] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=230, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:32.644499] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=11627183, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=230, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.649814] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.650178] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EAD07-0-0] [lt=27][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1001, ls_id:{id:1}, add_timestamp:1747300526118711}) [2025-05-15 09:40:32.659924] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.670039] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.681650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.691764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.701703] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.701873] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.702533] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.702568] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032702526) [2025-05-15 09:40:32.702584] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302032502480, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.702619] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12334950395, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.702639] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.702666] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.702682] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.702693] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.702709] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12334951451, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.702721] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.702735] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.702746] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.702755] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.702767] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.702777] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.702786] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032702598) [2025-05-15 09:40:32.711985] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.714523] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.714543] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.714559] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.714575] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.714588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.714600] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.714611] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.714622] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.714633] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302032714506, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.714648] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.714657] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.714716] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.714728] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.714735] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.714743] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.718356] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=19] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:32.719653] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.719672] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.719688] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.719704] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302032719568, register_time_us=-1, curr_time_us=1747302032719702) [2025-05-15 09:40:32.719724] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.719736] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.719752] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:32.720831] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:32.720854] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.720868] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.720883] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:32.720893] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:32.720901] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:32.722098] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.724240] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.724283] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032724234) [2025-05-15 09:40:32.724301] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302032524193, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.724331] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12334995963, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.724349] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.724373] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:32.724399] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.724410] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.724426] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.724436] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:32.724448] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032724315) [2025-05-15 09:40:32.724493] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.724514] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032724482) [2025-05-15 09:40:32.724534] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302032524433, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.724573] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=12334823470, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.724601] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.724632] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:32.724652] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.724666] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.724684] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.724695] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:32.724702] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302032724557) [2025-05-15 09:40:32.731573] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:32.731593] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.731602] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.732211] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.741346] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3956][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:32.742335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.744626] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.744651] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11527072, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.744667] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11527072, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.744690] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.744703] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.744714] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.744728] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.744743] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032744742, replica_locations:[]}) [2025-05-15 09:40:32.744760] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.744777] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:230, local_retry_times:230, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:32.744798] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.744817] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.744826] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.744837] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:32.744868] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:32.746034] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.746062] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.746097] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.746112] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.746125] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.746147] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.746159] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.746167] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.746181] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.746192] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032746192, replica_locations:[]}) [2025-05-15 09:40:32.746208] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.746221] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.746233] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.746248] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:32.746258] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:32.746268] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:32.746285] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:32.746297] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:32.746307] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:32.746314] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:32.746322] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:32.746331] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:32.746343] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:32.746353] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:32.746362] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:32.746371] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:32.746380] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:32.746390] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:32.746399] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:32.746415] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:32.746425] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.746435] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:32.746446] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:32.746457] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:32.746466] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=231, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:32.746500] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] will sleep(sleep_us=100000, remain_us=11525182, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=231, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.746965] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [4460][T1001_TimerWK0_TntSharedTimer][T1001][Y0-0000000000000000-0-0] [lt=22] clear dtl interm result cost(us)(clear_cost=24, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-15 09:40:32.752447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.753645] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.753663] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:32.753682] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:32.753701] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:32.753714] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=10398720) [2025-05-15 09:40:32.753744] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=30, size_used=0, mem_used=0) [2025-05-15 09:40:32.753783] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:32.756436] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=18] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts=0) [2025-05-15 09:40:32.756455] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [4460][T1001_TimerWK0_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=18] update reserved snapshot finished(cost_ts=21, reserved_snapshot=0) [2025-05-15 09:40:32.762565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.772627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.782732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.792840] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.801782] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:32.801801] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:32.801812] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.802605] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12335050830, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.802627] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.802649] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.802665] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:32.802675] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:32.802692] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12335050830, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.802704] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.802719] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:32.802731] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:32.802738] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:32.802747] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.802756] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:32.802771] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.802947] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.813057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.814611] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.814631] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.814651] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.814669] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.814687] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.814696] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.814704] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.814717] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302032814589, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.814732] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.814738] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.814796] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.814808] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.814816] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:32.814827] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:32.819708] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.819728] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.819742] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.819769] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=22] gc stale ls task succ [2025-05-15 09:40:32.820897] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:32.820919] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.820932] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.820947] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:32.820956] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:32.820967] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:32.823166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.824306] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12335096371, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.824328] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.824350] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.824564] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12334923878, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.824602] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=35] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.824621] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.827115] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4462][T1001_TimerWK2_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=14] ====== check clog disk timer task ====== [2025-05-15 09:40:32.827134] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4462][T1001_TimerWK2_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=16] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:32.831634] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:32.831651] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.831661] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.833278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.834467] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] Cache replace map node details(ret=0, replace_node_count=0, replace_time=3015, replace_start_pos=5800000, replace_num=100000) [2025-05-15 09:40:32.834497] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=10) [2025-05-15 09:40:32.843390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.846618] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.846640] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11425078, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.846655] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11425078, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.846679] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.846699] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.846710] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.846724] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.846739] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032846738, replica_locations:[]}) [2025-05-15 09:40:32.846757] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.846778] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:231, local_retry_times:231, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:32.846797] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.846815] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.846824] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.846832] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:32.846849] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:32.847912] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.847941] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.847972] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.847986] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.848001] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.848022] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.848035] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.848045] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.848058] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.848070] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032848070, replica_locations:[]}) [2025-05-15 09:40:32.848085] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.848097] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.848108] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.848124] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:32.848133] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:32.848142] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:32.848159] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:32.848171] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:32.848180] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:32.848194] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:32.848206] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:32.848214] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:32.848226] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:32.848236] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:32.848243] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:32.848252] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:32.848259] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:32.848268] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:32.848278] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:32.848291] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:32.848301] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.848311] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:32.848320] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:32.848332] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:32.848340] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=232, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:32.848362] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] will sleep(sleep_us=100000, remain_us=11423319, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=232, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.853498] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.855937] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=19] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:32.863610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.866298] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=49][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.866326] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.866338] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:32.866365] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:32.867406] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302032867403, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-05-15 09:40:32.867473] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4460][T1001_TimerWK0_Flush][T1001][Y0-0000000000000000-0-0] [lt=7] ====== traversal_flush timer task ====== [2025-05-15 09:40:32.867673] INFO [MDS] flush (mds_table_mgr.cpp:184) [4460][T1001_TimerWK0_Flush][T1001][YB427F000001-000635291CFC6455-0-0] [lt=36] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, {tablet_id:{id:60157}, rec_scn:{val:1747274416328312026, v:0}}, {tablet_id:{id:403}, rec_scn:{val:1747274416328312027, v:0}}, {tablet_id:{id:101079}, rec_scn:{val:1747274416328312028, v:0}}, {tablet_id:{id:50307}, rec_scn:{val:1747274416328312029, v:0}}, {tablet_id:{id:50272}, rec_scn:{val:1747274416328312030, v:0}}, {tablet_id:{id:383}, rec_scn:{val:1747274416328312031, v:0}}, {tablet_id:{id:238}, rec_scn:{val:1747274416328312032, v:0}}, {tablet_id:{id:60306}, rec_scn:{val:1747274416328312033, v:0}}, {tablet_id:{id:50333}, rec_scn:{val:1747274416328312034, v:0}}, {tablet_id:{id:150}, rec_scn:{val:1747274416328312035, v:0}}, {tablet_id:{id:50238}, rec_scn:{val:1747274416328312036, v:0}}, {tablet_id:{id:406}, rec_scn:{val:1747274416328312037, v:0}}, {tablet_id:{id:50531}, rec_scn:{val:1747274416328312038, v:0}}, {tablet_id:{id:1}, rec_scn:{val:1747274416328312039, v:0}}, {tablet_id:{id:60330}, rec_scn:{val:1747274416328312040, v:0}}, {tablet_id:{id:50319}, rec_scn:{val:1747274416328312041, v:0}}, {tablet_id:{id:50377}, rec_scn:{val:1747274416328312042, v:0}}, {tablet_id:{id:396}, rec_scn:{val:1747274416328312043, v:0}}, {tablet_id:{id:467}, rec_scn:{val:1747274416328312044, v:0}}, {tablet_id:{id:60407}, rec_scn:{val:1747274416328312045, v:0}}, {tablet_id:{id:50430}, rec_scn:{val:1747274416328312046, v:0}}, {tablet_id:{id:50104}, rec_scn:{val:1747274416328312047, v:0}}, {tablet_id:{id:50425}, rec_scn:{val:1747274416328312048, v:0}}, {tablet_id:{id:50480}, rec_scn:{val:1747274416328312049, v:0}}, {tablet_id:{id:60503}, rec_scn:{val:1747274416328312050, v:0}}, {tablet_id:{id:60390}, rec_scn:{val:1747274416328312051, v:0}}, {tablet_id:{id:50436}, rec_scn:{val:1747274416328312052, v:0}}, {tablet_id:{id:50160}, rec_scn:{val:1747274416328312053, v:0}}, {tablet_id:{id:474}, rec_scn:{val:1747274416328312054, v:0}}, {tablet_id:{id:60393}, rec_scn:{val:1747274416328312055, v:0}}, {tablet_id:{id:50451}, rec_scn:{val:1747274416328312056, v:0}}, }, max_consequent_callbacked_scn={val:1747289697900328000, v:0}, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:32.867841] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4460][T1001_TimerWK0_Flush][T1001][YB427F000001-000635291CFC6455-0-0] [lt=163] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-15 09:40:32.873286] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=32] PNIO [ratelimit] time: 1747302032873283, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:32.873717] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.883827] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.893935] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.896182] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=30] ====== tenant freeze timer task ====== [2025-05-15 09:40:32.901815] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:32.902651] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.902676] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032902644) [2025-05-15 09:40:32.902688] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302032702596, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.902709] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1002, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:32.902742] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12335151287, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.902758] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.902790] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12335151287, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:32.902804] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.904042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.910618] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:32.910624] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:32.912785] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:32.912807] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.912817] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:32.912934] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=109][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:32.914150] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.914660] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.914681] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:32.914699] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:32.914717] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.914730] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:32.914742] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.914755] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:32.914765] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:32.914776] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302032914639, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.914791] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:32.914798] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:32.914859] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:32.914870] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:32.918620] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:32.919766] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.919785] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.919796] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.920917] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:32.920937] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.921034] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=95][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.921049] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:32.921059] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:32.921069] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:32.924260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.924359] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.924385] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032924353) [2025-05-15 09:40:32.924401] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302032724315, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.924433] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12335195797, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:32.924453] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.924602] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:32.924623] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302032924599) [2025-05-15 09:40:32.924637] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302032724556, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:32.924661] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12335023304, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:32.924681] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:32.931694] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=72][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:32.931713] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:32.931723] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:32.933292] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4818638437, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:32.933384] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=28] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:32.933400] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4818638437, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:32.934370] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.934729] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.934750] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.934767] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.934781] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.934800] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.934813] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.934824] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.934838] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.934850] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.934863] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:32.934874] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:32.934883] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:32.934905] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.934916] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.934928] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.934945] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.934956] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.934966] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.934977] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.934991] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032934989, replica_locations:[]}) [2025-05-15 09:40:32.935015] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=23][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.935033] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:32.935044] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.935058] WDIAG [SERVER] refresh_sys_tenant_ls (ob_service.cpp:2711) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4721] fail to refresh sys tenant log stream(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, SYS_LS={id:1}) [2025-05-15 09:40:32.935069] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:32.935115] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:32.935129] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.935162] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=33] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:32.935182] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=19][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:32.935204] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=21] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=106, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:32.935226] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:32.935248] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=20][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:32.935257] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:32.944480] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.948474] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.948519] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=42][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11323216, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.948565] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=46][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11323216, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.948577] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.948593] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.948606] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.948616] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.948629] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.948642] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032948641, replica_locations:[]}) [2025-05-15 09:40:32.948657] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:32.948675] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:232, local_retry_times:232, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:32.948695] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:32.948712] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.948721] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:32.948729] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:32.948748] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:32.949906] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:32.949957] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:32.949971] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.949984] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:32.950003] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.950017] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:32.950025] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:32.950038] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:32.950050] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302032950049, replica_locations:[]}) [2025-05-15 09:40:32.950112] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11321570, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=233, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:32.954599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.958994] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:32.959026] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:32.959043] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:32.959058] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:32.959073] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:32.959084] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:32.959101] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:32.959113] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:32.959124] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:32.959131] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=6][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:32.959139] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:32.959150] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:32.959162] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:32.959173] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:32.959193] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:32.959208] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=14][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.959227] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=17][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:32.959249] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=20][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:32.959273] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=21][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:32.959289] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:32.959301] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:32.959323] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:32.959344] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:32.959356] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:32.959366] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:32.959387] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:32.959401] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4535][T1002_Occam][T1][YB427F000001-000635291FAC623F-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.959414] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4535][T1002_Occam][T1002][YB427F000001-000635291FAC623F-0-0] [lt=12][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:32.959427] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:32.959441] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:32.959452] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:32.959463] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302032958741, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:32.959475] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:32.959497] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, tenant_id=1, columns=0x7f1a84e563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:32.959519] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:32.959607] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1a84e563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:32.959624] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:32.959638] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:32.959649] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:32.959669] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:32.964708] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.969603] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:44, tid:3961}, {errcode:-4023, dropped:34, tid:4630}]) [2025-05-15 09:40:32.974819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.984930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.995042] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:32.995517] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD8-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.996814] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CD9-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:32.997696] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.997731] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:32.997740] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:32.997754] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:32.997762] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:32.997777] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:32.997796] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:32.997813] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:32.997848] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:32.997855] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:32.998439] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:33.001873] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.002729] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12335250748, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.002757] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.002793] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.002812] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.002823] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.002839] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12335250748, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.002851] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.002865] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.002876] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.002884] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.002895] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.002904] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.002914] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033002703) [2025-05-15 09:40:33.005158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.007571] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=72791040) [2025-05-15 09:40:33.007657] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=1] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:33.014743] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.015269] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.019827] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.019848] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.019860] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.020972] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.024437] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12335296298, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.024463] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.024498] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.024517] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.024530] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.024546] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.024559] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.024569] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033024419) [2025-05-15 09:40:33.024664] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.024699] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033024659) [2025-05-15 09:40:33.024717] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302032924651, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.024739] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12335123805, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.024754] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.024772] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.024786] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.024799] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.024813] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.024823] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.024830] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033024730) [2025-05-15 09:40:33.025382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.031751] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=106][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.034577] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=9] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=9) [2025-05-15 09:40:33.035499] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.035984] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4462][T1001_TimerWK2_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=9] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:33.036007] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4462][T1001_TimerWK2_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=16] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=79) [2025-05-15 09:40:33.045614] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.050244] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.050270] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11221458, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.050286] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11221458, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.050300] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.050310] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=11221458, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.050325] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.050342] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.050358] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.050372] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.050383] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.050397] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.050414] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033050412, replica_locations:[]}) [2025-05-15 09:40:33.050431] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.050456] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.050466] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:33.050514] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.050565] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11796941042, cache_obj->added_lc()=false, cache_obj->get_object_id()=15756, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.051834] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.051855] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.051870] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.051882] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.051893] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.051905] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.051919] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.051932] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.051945] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.051955] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.051969] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.051981] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033051981, replica_locations:[]}) [2025-05-15 09:40:33.052046] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11219636, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=234, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.055724] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.056760] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.056798] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:33.056811] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC967F-0-0] [lt=31][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302033056783}) [2025-05-15 09:40:33.056824] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:33.056836] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:33.056853] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:33.057230] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.057248] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:33.057263] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:33.064878] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552872, support_ssl=false) [2025-05-15 09:40:33.064949] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552872, sess.client_addr_="172.22.0.1:45498") [2025-05-15 09:40:33.064968] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552872, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.065167] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.065212] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.065249] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.065273] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.065291] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.065303] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.065329] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.065349] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552872, proxy_sessid=0) [2025-05-15 09:40:33.065391] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=12][errcode=0] server close connection(sessid=3221552872, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.065403] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552872, proxy_sessid=0) [2025-05-15 09:40:33.065411] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:33.065425] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9062-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552872, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.065496] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552872, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033064298, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.065524] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552872, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.065539] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552872, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033064298, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.065560] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552872) [2025-05-15 09:40:33.065582] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552872, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.065835] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.068820] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:33.068847] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.068860] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:33.068872] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:33.068883] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:33.068963] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=80][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:33.069815] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:42, tid:4504}]) [2025-05-15 09:40:33.075949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.084140] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4461][T1001_TimerWK1_][T1001][Y0-0000000000000000-0-0] [lt=11] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:33.084164] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4461][T1001_TimerWK1_][T1001][Y0-0000000000000000-0-0] [lt=19] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:33.084201] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4461][T1001_TimerWK1_][T1001][Y0-0000000000000000-0-0] [lt=35] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=63) [2025-05-15 09:40:33.086062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.096179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.101935] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.102769] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.102794] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033102762) [2025-05-15 09:40:33.102862] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=69][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302032902700, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.102895] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12335351105, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.102913] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.102936] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.102954] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.102964] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.102979] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12335351105, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.102992] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.103005] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.103017] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.103025] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.103038] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.103047] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.103057] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033102875) [2025-05-15 09:40:33.104222] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221552873, support_ssl=false) [2025-05-15 09:40:33.104302] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35] sm conn init succ(conn.sessid_=3221552873, sess.client_addr_="172.22.0.1:45512") [2025-05-15 09:40:33.104321] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552873, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.104514] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552874, support_ssl=false) [2025-05-15 09:40:33.104550] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552875, support_ssl=false) [2025-05-15 09:40:33.104556] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.104589] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=34] sm conn init succ(conn.sessid_=3221552874, sess.client_addr_="172.22.0.1:45528") [2025-05-15 09:40:33.104606] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.104610] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552874, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.104641] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.104684] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221552875, sess.client_addr_="172.22.0.1:45522") [2025-05-15 09:40:33.104682] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=38][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.104675] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552876, support_ssl=false) [2025-05-15 09:40:33.104700] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221552877, support_ssl=false) [2025-05-15 09:40:33.104706] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=24][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.104704] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552875, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.104729] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=22][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.104737] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552877, sess.client_addr_="172.22.0.1:45544") [2025-05-15 09:40:33.104741] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=34] sm conn init succ(conn.sessid_=3221552876, sess.client_addr_="172.22.0.1:45542") [2025-05-15 09:40:33.104759] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552877, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.104763] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=22] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.104760] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552876, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.104781] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552873, proxy_sessid=0) [2025-05-15 09:40:33.104798] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.104820] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=21][errcode=0] server close connection(sessid=3221552873, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.104834] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552873, proxy_sessid=0) [2025-05-15 09:40:33.104831] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.104844] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:33.104858] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9094-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552873, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.104869] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=39][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.104909] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=38][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.104916] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.104921] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.104933] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=23][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.104947] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.104947] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552873, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.104959] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.104961] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.104970] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552873, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.104976] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.104977] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.104980] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.104984] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552873, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.104990] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.104995] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.105001] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.105002] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552873) [2025-05-15 09:40:33.105002] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552874, proxy_sessid=0) [2025-05-15 09:40:33.105008] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.105009] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.105008] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.105021] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.105022] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552873, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.105034] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.105029] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105039] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105047] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552876, proxy_sessid=0) [2025-05-15 09:40:33.105046] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=18][errcode=0] server close connection(sessid=3221552874, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105051] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552875, proxy_sessid=0) [2025-05-15 09:40:33.105065] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552874, proxy_sessid=0) [2025-05-15 09:40:33.105076] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:33.105087] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=15][errcode=0] server close connection(sessid=3221552875, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105099] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552875, proxy_sessid=0) [2025-05-15 09:40:33.105108] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:33.105100] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=19][errcode=0] server close connection(sessid=3221552876, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105095] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9058-0-0] [lt=17] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552874, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.105117] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552876, proxy_sessid=0) [2025-05-15 09:40:33.105116] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9063-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552875, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.105137] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=20][errcode=-4016] session is null [2025-05-15 09:40:33.105146] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=29][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.105154] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552874, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.105149] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9095-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552876, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.105162] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552875, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.105177] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552874, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.105172] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.105190] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552875, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.105189] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552874, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.105193] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=21][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.105206] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552874) [2025-05-15 09:40:33.105201] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552875, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.105209] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.105220] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552875) [2025-05-15 09:40:33.105227] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105215] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552876, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.105228] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552874, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.105244] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552877, proxy_sessid=0) [2025-05-15 09:40:33.105246] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552875, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.105263] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=43] kill and revert session(conn.sessid_=3221552876, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.105277] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=21][errcode=0] server close connection(sessid=3221552877, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105293] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552877, proxy_sessid=0) [2025-05-15 09:40:33.105280] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552876, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.105300] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:33.105305] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552876) [2025-05-15 09:40:33.105317] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9059-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552877, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.105334] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221552876, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.105385] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221552878, support_ssl=false) [2025-05-15 09:40:33.105379] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552877, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.105405] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552877, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.105446] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552877, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302033104415, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.105437] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552878, sess.client_addr_="172.22.0.1:45552") [2025-05-15 09:40:33.105468] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552877) [2025-05-15 09:40:33.105470] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552878, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.105482] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552877, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.105747] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=52] new mysql sessid created(conn.sessid_=3221552879, support_ssl=false) [2025-05-15 09:40:33.105813] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] sm conn init succ(conn.sessid_=3221552879, sess.client_addr_="172.22.0.1:45562") [2025-05-15 09:40:33.105831] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552879, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.105867] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.105908] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.105922] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.105937] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.105944] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.105954] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.105968] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.105980] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552878, proxy_sessid=0) [2025-05-15 09:40:33.105994] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.106002] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=11][errcode=0] server close connection(sessid=3221552878, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.106013] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552878, proxy_sessid=0) [2025-05-15 09:40:33.106018] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.106020] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:33.106031] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9064-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552878, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.106037] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=31][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.106058] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.106074] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.106081] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.106102] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.106097] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552878, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033105476, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.106118] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552879, proxy_sessid=0) [2025-05-15 09:40:33.106121] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552878, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.106130] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552878, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033105476, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.106138] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=7][errcode=0] server close connection(sessid=3221552879, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.106139] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552880, support_ssl=false) [2025-05-15 09:40:33.106148] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552878) [2025-05-15 09:40:33.106149] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552879, proxy_sessid=0) [2025-05-15 09:40:33.106157] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:33.106162] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552878, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.106165] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9096-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552879, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.106179] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552880, sess.client_addr_="172.22.0.1:45568") [2025-05-15 09:40:33.106192] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552880, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.106201] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552879, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033105476, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.106222] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552879, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.106244] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552879, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033105476, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.106272] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] mark session id unused(conn.sessid_=3221552879) [2025-05-15 09:40:33.106290] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552879, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.106298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.106343] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.106383] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.106425] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.106457] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=30][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.106476] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.106509] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=32][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.106532] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.106556] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=20][errcode=-4018] get session fail(ret=-4018, sessid=3221552880, proxy_sessid=0) [2025-05-15 09:40:33.106585] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=10][errcode=0] server close connection(sessid=3221552880, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.106612] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=26][errcode=-4018] get session fail(ret=-4018, sessid=3221552880, proxy_sessid=0) [2025-05-15 09:40:33.106639] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=26][errcode=-4016] session is null [2025-05-15 09:40:33.106650] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905A-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552880, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.106711] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552880, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033105476, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.106740] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221552880, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.106749] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552880, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033105476, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.106768] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552880) [2025-05-15 09:40:33.106783] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552880, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.107521] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552881, support_ssl=false) [2025-05-15 09:40:33.107569] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552881, sess.client_addr_="172.22.0.1:45578") [2025-05-15 09:40:33.107586] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552881, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.107596] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=13] ====== checkpoint timer task ====== [2025-05-15 09:40:33.107746] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.107779] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.107794] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.107815] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.107825] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.107843] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=17][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.107861] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.107880] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552881, proxy_sessid=0) [2025-05-15 09:40:33.107916] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=19][errcode=0] server close connection(sessid=3221552881, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.107934] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552881, proxy_sessid=0) [2025-05-15 09:40:33.107945] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:33.107976] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221552882, support_ssl=false) [2025-05-15 09:40:33.107963] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9065-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552881, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.108017] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552882, sess.client_addr_="172.22.0.1:45586") [2025-05-15 09:40:33.108036] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552882, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.108073] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552881, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033107597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.108096] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552881, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.108105] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552881, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033107597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.108125] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552881) [2025-05-15 09:40:33.108145] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552883, support_ssl=false) [2025-05-15 09:40:33.108137] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552881, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.108174] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.108186] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552883, sess.client_addr_="172.22.0.1:45594") [2025-05-15 09:40:33.108201] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.108199] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552883, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.108219] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.108244] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.108253] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.108269] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.108285] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.108303] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552882, proxy_sessid=0) [2025-05-15 09:40:33.108325] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=8][errcode=0] server close connection(sessid=3221552882, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.108340] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552882, proxy_sessid=0) [2025-05-15 09:40:33.108348] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:33.108348] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.108364] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9097-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552882, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.108379] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.108381] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=57] succ to get rec scn(*this={this:0x7f1a94c04030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300521088740, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1007}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:33.108406] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.108411] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552882, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302033107597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.108426] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.108432] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552882, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.108438] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.108425] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=44] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a95378080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a95378080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:33.108449] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.108442] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552882, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302033107597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.108461] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552882) [2025-05-15 09:40:33.108466] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.108475] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552882, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.108503] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221552883, proxy_sessid=0) [2025-05-15 09:40:33.108527] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=10][errcode=0] server close connection(sessid=3221552883, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.108541] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552883, proxy_sessid=0) [2025-05-15 09:40:33.108548] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:33.108556] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905B-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552883, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.108608] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552883, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033107597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.108630] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552883, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.108642] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552883, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033107597, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.108659] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552883) [2025-05-15 09:40:33.108671] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552883, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.108641] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=36] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1747274416328312025, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, }, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:33.108730] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=87] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1a979cc1b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1a979d7010, memtable_head:0, memtable_tail:1, t3m:0x7f1aaa0ea030, tables:[0x7f1a952dc080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1a979d8d10, ls_tablet_svr:0x7f1a979cc190}, min_rec_scn={val:1747274408663046000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:33.108769] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=33] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-15 09:40:33.108791] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=19] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:33.108807] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=15] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:33.108817] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=9] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:33.108842] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=19] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:33.108869] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=22] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:33.108889] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274408663046000, v:0}, checkpoint_scn_in_ls_meta={val:1747274408663046000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:33.108918] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:5167067136}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:5167067136}, lsn={lsn:5167067136}) [2025-05-15 09:40:33.108948] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=29] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:5167067136}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:33.108959] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4462][T1001_TimerWK2_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=10] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-15 09:40:33.114835] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.114858] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.114865] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:33.114884] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:33.114880] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.114892] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=8] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:33.114907] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.114910] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=17] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:33.114925] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.114939] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.114957] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.114966] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.114980] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302033114808, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.114998] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.115007] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.115084] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.115099] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.115108] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.115122] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.116440] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=38][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.118885] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:33.119893] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.119918] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.119934] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.119992] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:33.124482] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.124532] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033124476) [2025-05-15 09:40:33.124551] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302032924416, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.124579] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12335396657, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.124602] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.124631] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.124654] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.124664] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.124689] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.124697] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.124705] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033124563) [2025-05-15 09:40:33.124730] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12335224164, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.124751] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.124766] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.124784] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.124792] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.124803] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.124818] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.124825] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033124717) [2025-05-15 09:40:33.126586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.127778] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:33.127821] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=36] temporary file module use no blocks [2025-05-15 09:40:33.127850] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=28] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:33.127879] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=25] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:33.127901] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11796018196, flush_io_finished_ret_:0, flush_io_finished_round_:1507, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:33.136705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.146821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.152209] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.152237] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11119497, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.152258] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11119497, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.152278] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.152290] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=11119497, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.152309] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.152332] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.152354] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.152375] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.152384] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.152404] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.152427] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033152425, replica_locations:[]}) [2025-05-15 09:40:33.152452] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.152483] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.152513] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:33.152545] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.152611] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11797043084, cache_obj->added_lc()=false, cache_obj->get_object_id()=15757, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.154025] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.154049] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.154065] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.154081] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.154091] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.154115] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.154131] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.154149] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.154167] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.154176] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.154197] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.154216] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033154215, replica_locations:[]}) [2025-05-15 09:40:33.154295] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11117387, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=235, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.156936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.157462] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4586][T1002_DupTbLeas][T1001][YB427F000001-00063529201C618F-0-0] [lt=22][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.157486] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4586][T1002_DupTbLeas][T1001][YB427F000001-00063529201C618F-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.157520] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4586][T1002_DupTbLeas][T1001][YB427F000001-00063529201C618F-0-0] [lt=23][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%" order by ls_id limit 1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.157541] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4586][T1002_DupTbLeas][T1001][YB427F000001-00063529201C618F-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.157558] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4586][T1002_DupTbLeas][T1002][YB427F000001-00063529201C618F-0-0] [lt=16][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%" order by ls_id limit 1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.157567] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select * from __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:33.157595] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.157604] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.157620] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302033157438, sql=select * from __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:33.157639] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.157649] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select * from __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:33.157758] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:663) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] fail to inner get ls status info(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=select * from __all_ls_status where tenant_id = 1002 and flag like "%DUPLICATE%" order by ls_id limit 1, tenant_id=1002, exec_tenant_id=1001, need_member_list=false) [2025-05-15 09:40:33.157780] WDIAG [STORAGE.DUP_TABLE] refresh_dup_ls_ (ob_dup_table_util.cpp:110) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get duplicate ls status info failed(ret=-5627, tmp_dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-05-15 09:40:33.157805] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:202) [4586][T1002_DupTbLeas][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=0] refresh dup ls failed(tmp_ret=-5627, this={tenant_id_:1002, dup_table_scan_timer_:0x7f1a8a43dae8, dup_loop_worker_:0x7f1a8a7066f0, min_dup_ls_status_info_:{tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, tenant_schema_dup_tablet_set_.size():0, scan_task_execute_interval_:10000000, last_dup_ls_refresh_time_:0, last_dup_schema_refresh_time_:0, last_scan_task_succ_time_:1747302021153444, max_execute_interval_:12002741}) [2025-05-15 09:40:33.159029] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=20] ====== check clog disk timer task ====== [2025-05-15 09:40:33.159056] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=23] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:33.167058] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.174478] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.174511] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.174791] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.174885] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.177175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.182106] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:33.182139] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:33.182154] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:33.182172] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:33.182184] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:33.182199] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:33.182213] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:33.182228] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=14][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:33.182237] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:33.182252] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=14][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:33.182261] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:33.182270] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:33.182286] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=14][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:33.182294] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:33.182347] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=43][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:33.182363] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=15][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.182374] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.182390] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=14][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:33.182400] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:33.182417] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:33.182426] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:33.182450] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:33.182474] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.182500] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=25][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.182506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:33.182531] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=16][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:33.182549] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646E-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.182559] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:33.182575] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=15][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.182584] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:33.182599] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=14][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:33.182607] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=8][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302033181850, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.182625] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=18][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:33.182635] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.182724] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=15][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:33.182741] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=16][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:33.182752] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:33.182769] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=16][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:33.182793] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=19][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:33.182810] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=16][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:33.182819] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646E-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:33.187290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.191256] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:33.195349] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4580][T1002_TimerWK3_TenantMetaMemMg][T1002][Y0-0000000000000000-0-0] [lt=0] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:728, free_obj_hold_cnt:0, allocator used:2795520, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=3002880, tablets_mem_limit=7774753240, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:9, free_obj_hold_cnt:0, allocator used:18432, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=728) [2025-05-15 09:40:33.197409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.202000] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.202847] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12335451548, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.202878] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.202908] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.202930] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.202940] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.202961] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1001}, delta=12335451548, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.202972] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.203012] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.203022] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.203062] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.203072] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.203087] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.203096] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033202823) [2025-05-15 09:40:33.207521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.214890] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.214914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.214932] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.214970] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=37][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.214982] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.214998] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.215007] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.215018] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.215032] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302033214868, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.215044] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.215059] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.215115] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.215130] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.215139] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.215156] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.217635] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.219953] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.219979] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.219992] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.224561] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12335496048, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.224591] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.224621] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.224643] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.224655] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.224675] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.224691] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.224698] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033224537) [2025-05-15 09:40:33.224783] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.224811] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033224777) [2025-05-15 09:40:33.224831] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302033024730, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.224853] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12335323555, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.224872] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.224893] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.224904] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.224919] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.224930] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.224944] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.224952] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033224843) [2025-05-15 09:40:33.228117] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.234669] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:33.238777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.248892] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.249830] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=11][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:33.249855] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=24][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.249872] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=10][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.249981] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=110][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.249994] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.250012] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=15][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.250026] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.250048] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.250071] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.250091] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.250101] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.250121] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.250155] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=16] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.250173] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=16] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:33.251713] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=16] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:33.251742] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:33.251755] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:33.251773] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:33.251784] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:33.251799] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:33.251812] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:33.251826] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:33.251834] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:33.251848] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=13][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:33.251856] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:33.251872] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:33.251881] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:33.251894] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=14][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:33.251913] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:33.251927] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.251938] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.251954] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=14][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:33.251964] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:33.251982] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=16][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:33.251990] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:33.252013] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:33.252037] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.252044] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.252057] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=13][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:33.252071] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:33.252088] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.252097] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=8][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:33.252114] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=16][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.252135] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=20][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:33.252157] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=21][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:33.252179] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=20][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca3b0, start=1747302033251477, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.252202] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=23][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:33.252227] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=23][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.252313] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=25][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:33.252331] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=17][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:33.252340] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=9][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:33.252355] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=14][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:33.252370] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=12][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:33.252391] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=20][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:33.252417] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63BD-0-0] [lt=26][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:33.253834] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.253853] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:33.253862] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:33.253876] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:33.253895] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=6239232) [2025-05-15 09:40:33.253937] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:33.254431] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.254459] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11017269, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.254480] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=11017269, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.254511] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.254521] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=11017269, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.254539] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.254560] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.254580] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.254599] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.254608] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.254628] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.254645] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033254642, replica_locations:[]}) [2025-05-15 09:40:33.254666] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.254694] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.254710] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:33.254749] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.254800] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11797145278, cache_obj->added_lc()=false, cache_obj->get_object_id()=15758, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.256039] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.256064] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.256079] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.256097] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.256106] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.256124] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.256136] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.256154] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.256172] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.256181] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.256200] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.256211] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033256210, replica_locations:[]}) [2025-05-15 09:40:33.256286] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=11015395, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=236, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.258341] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:33.258374] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-05-15 09:40:33.258386] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:33.258409] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:33.258430] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11796148764, flush_io_finished_ret_:0, flush_io_finished_round_:1507, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:33.259007] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.265911] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:33.269126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.276199] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643E-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.276222] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643E-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.276239] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643E-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.276258] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643E-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.276268] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643E-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.276290] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:33.276301] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.276315] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.276327] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302033276185, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:33.276343] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.276351] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:33.276431] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=42][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1001, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:33.276479] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:33.277767] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643F-0-0] [lt=21][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.277788] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643F-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.277801] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643F-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.277818] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643F-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.277826] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C643F-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.277842] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:33.277850] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.277864] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.277872] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302033277759, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:33.277887] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.277895] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:33.277939] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] fail to select for update snapshot_gc_scn(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001) [2025-05-15 09:40:33.277956] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] failed to load updated info(ret=-5627) [2025-05-15 09:40:33.277965] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4461][T1001_TimerWK1_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to try update info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:33.279239] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.289353] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.296978] WDIAG load_file_to_string (utility.h:673) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:33.297003] WDIAG get_ethernet_speed (utility.cpp:575) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:33.297014] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:33.297030] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:33.299464] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.302056] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.302885] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001) [2025-05-15 09:40:33.302907] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.302942] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033302880) [2025-05-15 09:40:33.302961] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302033102874, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.302989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12335551037, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.303012] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.303038] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.303052] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.303059] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.303073] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12335551037, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.303091] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.303110] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.303122] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.303140] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.303157] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.303171] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.303179] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033302973) [2025-05-15 09:40:33.309586] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.314756] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.314778] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.314792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.314810] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.314819] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.314836] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.314844] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.314858] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.314866] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302033314742, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.314882] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.314890] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.314941] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.314958] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.314966] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.314981] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.319176] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=39] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:33.319700] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.320010] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.320036] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.320050] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.324599] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.324629] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033324593) [2025-05-15 09:40:33.324644] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302033124562, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.324733] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69] get wrs ts(ls_id={id:1}, delta=12335596715, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.324762] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.324792] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.324814] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.324824] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.324844] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.324859] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.324852] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12335424222, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.324868] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033324713) [2025-05-15 09:40:33.324871] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.324892] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.324910] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.324918] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.324937] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.324945] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.324960] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033324840) [2025-05-15 09:40:33.329814] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.339927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.350041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.353481] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:33.353526] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.353535] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:33.353544] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:33.353559] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:33.353641] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:33.353651] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.353694] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:33.353701] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:33.353708] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:33.353763] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:33.356414] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.356443] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10915284, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.356463] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10915284, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.356476] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.356499] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=10915284, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.356512] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.356538] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.356557] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.356577] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.356586] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.356606] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.356627] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033356625, replica_locations:[]}) [2025-05-15 09:40:33.356649] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.356678] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.356694] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:33.356720] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.356774] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11797247249, cache_obj->added_lc()=false, cache_obj->get_object_id()=15759, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.357925] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.357950] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.357969] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.357980] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.357995] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.358006] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.358025] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.358036] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.358054] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.358063] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.358081] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.358093] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033358092, replica_locations:[]}) [2025-05-15 09:40:33.358159] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10913522, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=237, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.359149] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:33.359223] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=22] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:33.359302] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=23] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:33.359366] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:33.359401] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:15141, total:15150) [2025-05-15 09:40:33.359442] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:33.359508] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=61] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:15141, total:15150) [2025-05-15 09:40:33.359533] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:33.359558] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:33.359576] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:33.359594] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628768][2nd=1895]}) [2025-05-15 09:40:33.360158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.360799] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=23] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-05-15 09:40:33.362256] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=20][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.362283] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=26][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.362292] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.362301] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=9][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.362317] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=14][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:33.362337] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=12][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:33.362355] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.362365] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_ReqMemEvict][T1002][YB427F000001-000635291F6C6674-0-0] [lt=9][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5157, ret=-5157) [2025-05-15 09:40:33.362383] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:33.362402] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.362411] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:33.362427] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78cada0, start=1747302033362234, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:33.362439] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:33.362454] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5157) [2025-05-15 09:40:33.362523] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] failed to fetch max rule version(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.362540] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] failed to fetch max rule version(ret=-5157) [2025-05-15 09:40:33.362548] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] failed to sync rule from inner table(ret=-5157) [2025-05-15 09:40:33.370274] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.373413] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.380392] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.380734] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [3960][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=26] log compressor cycles once. (ret=0, cost_time=1055, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=3626813415424, fast_delete_log_mode=false) [2025-05-15 09:40:33.383535] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-15 09:40:33.383539] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4291][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=1] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4820014452, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:33.383561] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=22] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-15 09:40:33.383592] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=27] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:33.383612] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=19] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:33.383627] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:33.383828] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4291][T1_TimerWK2][T1][Y0-0000000000000000-0-0] [lt=27] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4820014452, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:33.385192] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=28] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2025-05-15 09:40:33.385225] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2025-05-15 09:40:33.385241] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2025-05-15 09:40:33.385252] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2025-05-15 09:40:33.385264] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:33.385272] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=8][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:33.385288] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2025-05-15 09:40:33.385300] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:33.385312] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:33.385325] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:33.385336] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:33.385347] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:33.385358] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:33.385389] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=31][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:33.385412] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:33.385424] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.385437] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.385450] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:33.385460] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:33.385474] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:33.385481] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:33.385508] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=23][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:33.385528] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.385538] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.385544] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:33.385560] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:33.385573] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.385583] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63BE-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2025-05-15 09:40:33.385595] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:33.385605] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:33.385616] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:33.385626] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302033384952, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:33.385638] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:33.385645] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:33.385714] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2025-05-15 09:40:33.385725] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] failed to fetch max rule version(ret=-5019) [2025-05-15 09:40:33.385732] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2025-05-15 09:40:33.386252] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=125] Dumping All Cache Objs(alloc_obj_list.count()=1, alloc_obj_list=[{obj_id:15760, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:1, added_to_lc:false, mem_used:25832}]) [2025-05-15 09:40:33.386278] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=24] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:33.390512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.395159] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.395606] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.395606] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=27] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.396142] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.396364] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.396472] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.396556] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=42] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:33.400628] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.402112] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.402972] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12335651651, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.402996] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.403019] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.403036] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.403047] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.403062] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12335651651, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.403074] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.403088] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.403100] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.403109] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.403121] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.403130] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.403138] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033402950) [2025-05-15 09:40:33.411741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.413021] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.414830] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.414850] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.414863] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.414878] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.414892] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.414904] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.414915] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.414922] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.414933] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302033414816, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.414945] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.414952] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.415001] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.415013] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.415023] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.415033] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.420063] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.420084] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.420097] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.421871] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.424672] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12335696187, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.424697] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.424723] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.424740] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.424816] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=76][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.424833] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.424844] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.424851] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033424653) [2025-05-15 09:40:33.424901] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.424926] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033424895) [2025-05-15 09:40:33.424943] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302033224842, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.424968] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12335523694, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.424983] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.425001] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.425016] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.425026] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.425040] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.425050] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.425056] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033424957) [2025-05-15 09:40:33.431981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.434762] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=23] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:33.442091] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.452199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.458291] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.458314] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10813408, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.458346] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=32][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10813408, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.458360] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.458372] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=10813408, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.458385] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.458401] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.458416] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.458430] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.458440] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.458454] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.458470] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033458468, replica_locations:[]}) [2025-05-15 09:40:33.458498] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.458526] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.458538] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:33.458584] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.458634] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11797349112, cache_obj->added_lc()=false, cache_obj->get_object_id()=15760, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.459864] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.459885] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.459900] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.459912] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.459922] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.459934] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.459949] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.459962] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.459975] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.459983] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.459997] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.460010] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033460009, replica_locations:[]}) [2025-05-15 09:40:33.460077] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10811605, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=238, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.462308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.472415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.482521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.492627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.497898] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.497926] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.497935] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.498030] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=97][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.498038] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:33.498053] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:33.498069] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:33.498079] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:33.498114] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:33.498121] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.502166] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.502734] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.503010] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.503035] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033503002) [2025-05-15 09:40:33.503050] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302033302972, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.503081] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12335751127, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.503098] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.503120] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.503136] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.503144] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.503159] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12335751127, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.503171] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.503185] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.503195] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.503204] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.503216] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.503225] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.503234] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033503063) [2025-05-15 09:40:33.503696] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C6601-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.503718] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C6601-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.503740] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C6601-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.503758] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_FreInfoReload][T1001][YB427F000001-000635291F7C6601-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.503772] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6601-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.503781] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:33.503795] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.503804] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.503817] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302033503666, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:33.503833] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.503908] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=71][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:33.504001] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1002, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:33.504052] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:33.505426] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=12][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.505452] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=25][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.505462] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.505476] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.505494] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=17][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:33.505510] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=11][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:33.505526] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.505540] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_FreInfoReload][T1002][YB427F000001-000635291F7C6602-0-0] [lt=13][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5157, ret=-5157) [2025-05-15 09:40:33.505552] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:33.505564] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.505576] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:33.505587] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78cb440, start=1747302033505409, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:33.505601] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:33.505608] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-5157] execute sql failed(ret=-5157, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:33.505647] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to select for update snapshot_gc_scn(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.505660] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] failed to load updated info(ret=-5157) [2025-05-15 09:40:33.505668] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to try update info(tmp_ret=-5157, tmp_ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:33.512843] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.514841] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.514860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.514872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.514969] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=97][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.514981] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.514993] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.515004] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.515013] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.515024] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302033514830, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.515036] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.515047] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.515089] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.515100] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.515108] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.515121] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.517516] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:33.517555] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:33.519441] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=31] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:33.520120] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.520141] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.520157] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.522958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.524717] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.524742] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033524712) [2025-05-15 09:40:33.524757] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302033324712, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.524792] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12335796752, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.524810] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.524834] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.524852] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.524863] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.524877] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.524888] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.524898] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033524775) [2025-05-15 09:40:33.524964] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12335624259, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.524982] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.525000] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.525016] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.525026] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.525040] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.525027] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=27] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2025-05-15 09:40:33.525047] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.525062] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033524952) [2025-05-15 09:40:33.525064] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=35][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2025-05-15 09:40:33.525089] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=24][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2025-05-15 09:40:33.525109] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=19][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2025-05-15 09:40:33.525122] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:33.525137] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=16][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:33.525151] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2025-05-15 09:40:33.525167] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=15][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:33.525176] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:33.525190] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=14][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:33.525198] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:33.525212] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:33.525220] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:33.525234] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=13][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:33.525252] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:33.525266] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.525277] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.525293] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=15][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:33.525304] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1, ret=-5019) [2025-05-15 09:40:33.525321] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=16][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:33.525329] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:33.525352] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:33.525376] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=20][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.525384] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:33.525397] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=13][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:33.525425] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:33.525444] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4469][T1001_DupTbLeas][T1][YB427F000001-00063529200C618F-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.525455] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4469][T1001_DupTbLeas][T1001][YB427F000001-00063529200C618F-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1"}, aret=-5019, ret=-5019) [2025-05-15 09:40:33.525471] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:33.525480] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:33.525505] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:33.525515] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302033524825, sql=select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:33.525531] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:33.525540] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:33.525636] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:663) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select * from __all_ls_status where tenant_id = 1001 and flag like "%DUPLICATE%" order by ls_id limit 1, tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-05-15 09:40:33.525656] WDIAG [STORAGE.DUP_TABLE] refresh_dup_ls_ (ob_dup_table_util.cpp:110) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] get duplicate ls status info failed(ret=-5019, tmp_dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-05-15 09:40:33.525679] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:202) [4469][T1001_DupTbLeas][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=0] refresh dup ls failed(tmp_ret=-5019, this={tenant_id_:1001, dup_table_scan_timer_:0x7f1aa483dae8, dup_loop_worker_:0x7f1aa4b066f0, min_dup_ls_status_info_:{tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, tenant_schema_dup_tablet_set_.size():0, scan_task_execute_interval_:10000000, last_dup_ls_refresh_time_:0, last_dup_schema_refresh_time_:0, last_scan_task_succ_time_:1747302021520176, max_execute_interval_:12003674}) [2025-05-15 09:40:33.533068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.539618] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:33.541826] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=16] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:33.543181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.553298] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.557698] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:33.557724] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=9115, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:33.557740] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:33.557763] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:33.557777] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:33.557795] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:33.558162] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.558175] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:33.558191] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:33.559640] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [4460][T1001_TimerWK0_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=18] BlockGCTimerTask success(ret=0, cost_time_us=10, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"127.0.0.1:2882", log_dir:"/root/ob/observer/store/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):201, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-15 09:40:33.560206] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.560233] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10711494, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.560249] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10711494, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.560260] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.560269] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=10711494, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.560280] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.560295] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.560310] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.560324] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.560336] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.560351] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.560368] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033560366, replica_locations:[]}) [2025-05-15 09:40:33.560388] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.560413] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.560424] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:33.560452] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.560517] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11797450992, cache_obj->added_lc()=false, cache_obj->get_object_id()=15761, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.561759] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.561781] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.561797] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.561810] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.561819] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.561832] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.561847] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.561870] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.561885] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.561897] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.561912] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.561927] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033561926, replica_locations:[]}) [2025-05-15 09:40:33.561996] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10709686, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=239, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.563410] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.566418] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221552900, support_ssl=false) [2025-05-15 09:40:33.566515] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=39] sm conn init succ(conn.sessid_=3221552900, sess.client_addr_="172.22.0.1:45600") [2025-05-15 09:40:33.566536] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552900, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.566729] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.566771] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.566806] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.566828] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.566840] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.566851] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.566876] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.566896] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552900, proxy_sessid=0) [2025-05-15 09:40:33.566937] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=11][errcode=0] server close connection(sessid=3221552900, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.566950] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552900, proxy_sessid=0) [2025-05-15 09:40:33.566964] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:33.566975] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9066-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552900, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.567001] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6675-0-0] [lt=22][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.567020] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6675-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.567029] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552900, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033565872, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.567043] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6675-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.567056] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552900, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.567058] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6675-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.567072] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.567073] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552900, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033565872, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.567082] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.567094] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.567095] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552900) [2025-05-15 09:40:33.567105] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.567119] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302033566981, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.567134] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.567126] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552900, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.567146] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:33.567212] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=13][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:33.567227] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=13][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:33.567240] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=12][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:33.567251] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=10][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:33.567264] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=10][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:33.567277] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=11][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:33.567287] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6675-0-0] [lt=9][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:33.575673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.585787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.595896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.602226] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.603082] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12335851658, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.603108] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.603131] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.603153] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.603165] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.603181] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12335851658, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.603194] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.603209] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.603222] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.603235] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.603245] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.603253] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.603264] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033603063) [2025-05-15 09:40:33.606006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.606200] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552901, support_ssl=false) [2025-05-15 09:40:33.606296] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=45] sm conn init succ(conn.sessid_=3221552901, sess.client_addr_="172.22.0.1:45602") [2025-05-15 09:40:33.606317] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552901, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.606324] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=35] new mysql sessid created(conn.sessid_=3221552902, support_ssl=false) [2025-05-15 09:40:33.606339] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552903, support_ssl=false) [2025-05-15 09:40:33.606355] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221552904, support_ssl=false) [2025-05-15 09:40:33.606388] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552904, sess.client_addr_="172.22.0.1:45620") [2025-05-15 09:40:33.606395] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221552903, sess.client_addr_="172.22.0.1:45636") [2025-05-15 09:40:33.606404] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de16030, session_id:3221552904, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.606405] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221552902, sess.client_addr_="172.22.0.1:45618") [2025-05-15 09:40:33.606412] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552903, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.606430] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221552902, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.606473] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552905, support_ssl=false) [2025-05-15 09:40:33.606528] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221552905, sess.client_addr_="172.22.0.1:45646") [2025-05-15 09:40:33.606546] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de16db0, session_id:3221552905, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.606597] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.606646] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.606638] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=36] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.606662] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.606672] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.606696] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.606697] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.606696] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.606709] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.606713] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.606729] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.606737] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.606744] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.606739] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.606761] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606751] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.606773] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552902, proxy_sessid=0) [2025-05-15 09:40:33.606771] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=31][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.606787] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.606799] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=9][errcode=0] server close connection(sessid=3221552902, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606798] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.606809] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552902, proxy_sessid=0) [2025-05-15 09:40:33.606808] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=31] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.606817] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:33.606826] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=118][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.606828] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606826] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9067-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552902, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.606840] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.606844] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606846] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552903, proxy_sessid=0) [2025-05-15 09:40:33.606855] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552901, proxy_sessid=0) [2025-05-15 09:40:33.606857] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.606860] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.606865] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=7][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.606873] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.606879] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.606881] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=9][errcode=0] server close connection(sessid=3221552901, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606877] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221552902, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.606893] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606885] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=13][errcode=0] server close connection(sessid=3221552903, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606894] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552901, proxy_sessid=0) [2025-05-15 09:40:33.606903] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552904, proxy_sessid=0) [2025-05-15 09:40:33.606907] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:33.606905] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552902, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.606904] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552903, proxy_sessid=0) [2025-05-15 09:40:33.606918] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=7][errcode=0] server close connection(sessid=3221552904, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606921] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=16][errcode=-4016] session is null [2025-05-15 09:40:33.606915] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9098-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552901, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.606917] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221552902, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.606941] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221552902) [2025-05-15 09:40:33.606952] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.606938] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905C-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552903, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.606959] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=7][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.606967] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.606973] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.606973] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=55][errcode=-4018] get session fail(ret=-4018, sessid=3221552904, proxy_sessid=0) [2025-05-15 09:40:33.606961] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552902, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.606982] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:33.606983] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.606993] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552905, proxy_sessid=0) [2025-05-15 09:40:33.606989] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9068-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552904, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.606980] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552901, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607012] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=8][errcode=0] server close connection(sessid=3221552905, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.607001] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552903, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607028] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552905, proxy_sessid=0) [2025-05-15 09:40:33.607037] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:33.607027] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de16030, session_id:3221552904, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607038] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] kill and revert session(conn.sessid_=3221552903, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.607045] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC9099-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552905, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.607056] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552903, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.607056] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552901, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.607081] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552904, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.607083] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] mark session id unused(conn.sessid_=3221552903) [2025-05-15 09:40:33.607089] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552901, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.607113] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552901) [2025-05-15 09:40:33.607107] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552903, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.607136] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552901, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.607176] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] can close safely, do destroy(*s={this:0x7f1a1de16030, session_id:3221552904, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.607190] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552904) [2025-05-15 09:40:33.607204] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552904, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.607244] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de16db0, session_id:3221552905, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607278] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221552905, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.607290] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] new mysql sessid created(conn.sessid_=3221552906, support_ssl=false) [2025-05-15 09:40:33.607382] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552907, support_ssl=false) [2025-05-15 09:40:33.607380] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f1a1de16db0, session_id:3221552905, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302033606043, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.607405] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] mark session id unused(conn.sessid_=3221552905) [2025-05-15 09:40:33.607427] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552905, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.607434] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552907, sess.client_addr_="172.22.0.1:45662") [2025-05-15 09:40:33.607448] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=38] sm conn init succ(conn.sessid_=3221552906, sess.client_addr_="172.22.0.1:45648") [2025-05-15 09:40:33.607469] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552906, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607458] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552907, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607745] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=52] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.607776] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=42] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.607784] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.607805] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.607815] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.607823] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.607829] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.607834] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.607840] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.607842] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221552908, support_ssl=false) [2025-05-15 09:40:33.607843] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.607855] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.607853] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.607868] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.607870] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.607881] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552906, proxy_sessid=0) [2025-05-15 09:40:33.607885] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552907, proxy_sessid=0) [2025-05-15 09:40:33.607885] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552908, sess.client_addr_="172.22.0.1:45678") [2025-05-15 09:40:33.607900] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=9][errcode=0] server close connection(sessid=3221552906, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.607907] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=7][errcode=-4018] get session fail(ret=-4018, sessid=3221552906, proxy_sessid=0) [2025-05-15 09:40:33.607902] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de16030, session_id:3221552908, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607913] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=14][errcode=0] server close connection(sessid=3221552907, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.607919] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:33.607925] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552907, proxy_sessid=0) [2025-05-15 09:40:33.607928] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9069-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552906, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.607940] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:33.607948] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905D-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552907, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.607970] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552906, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033607102, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.607995] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552906, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.607995] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552907, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033607102, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.608014] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221552907, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.608009] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552906, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302033607102, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.608026] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552907, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033607102, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.608042] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552907) [2025-05-15 09:40:33.608027] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552906) [2025-05-15 09:40:33.608054] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552907, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.608083] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.608069] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31] connection close(sessid=3221552906, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.608104] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.608115] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.608128] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.608135] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.608141] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.608158] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.608170] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552908, proxy_sessid=0) [2025-05-15 09:40:33.608195] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=12][errcode=0] server close connection(sessid=3221552908, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.608210] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552908, proxy_sessid=0) [2025-05-15 09:40:33.608222] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:33.608228] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909A-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552908, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.608273] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de16030, session_id:3221552908, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302033607102, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.608293] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552908, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.608300] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de16030, session_id:3221552908, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302033607102, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.608312] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] mark session id unused(conn.sessid_=3221552908) [2025-05-15 09:40:33.608324] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552908, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.609033] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=34] new mysql sessid created(conn.sessid_=3221552909, support_ssl=false) [2025-05-15 09:40:33.609088] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552909, sess.client_addr_="172.22.0.1:45692") [2025-05-15 09:40:33.609106] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552909, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.609257] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.609291] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.609305] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.609317] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.609329] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.609335] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.609352] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.609364] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552909, proxy_sessid=0) [2025-05-15 09:40:33.609384] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=10][errcode=0] server close connection(sessid=3221552909, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.609394] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552909, proxy_sessid=0) [2025-05-15 09:40:33.609401] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:33.609411] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906A-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552909, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.609458] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552909, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033609214, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.609482] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221552909, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.609505] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552909, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302033609214, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.609523] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552909) [2025-05-15 09:40:33.609539] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552909, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.609551] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552910, support_ssl=false) [2025-05-15 09:40:33.609602] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552910, sess.client_addr_="172.22.0.1:45702") [2025-05-15 09:40:33.609620] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552910, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.609646] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552911, support_ssl=false) [2025-05-15 09:40:33.609672] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552911, sess.client_addr_="172.22.0.1:45704") [2025-05-15 09:40:33.609685] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221552911, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.609836] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.609869] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.609888] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:33.609892] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.609903] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:33.609915] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.609920] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:33.609924] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.609933] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.609932] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:33.609943] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:33.609950] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.609954] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:33.609964] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552910, proxy_sessid=0) [2025-05-15 09:40:33.609968] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.609980] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552911, proxy_sessid=0) [2025-05-15 09:40:33.609990] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=14][errcode=0] server close connection(sessid=3221552910, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.609997] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=7][errcode=0] server close connection(sessid=3221552911, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:33.610004] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552910, proxy_sessid=0) [2025-05-15 09:40:33.610008] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552911, proxy_sessid=0) [2025-05-15 09:40:33.610012] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:33.610017] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:33.610021] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905E-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552910, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.610027] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909B-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552911, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:33.610074] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552910, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033609214, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.610106] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221552911, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302033609214, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:33.610126] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221552910, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.610139] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221552911, proxy_sessid=0, ret=0) [2025-05-15 09:40:33.610154] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552910, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302033609214, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.610171] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552910) [2025-05-15 09:40:33.610188] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552910, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.610210] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221552911, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302033609214, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:33.610226] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552911) [2025-05-15 09:40:33.610236] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552911, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:33.615048] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.615070] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.615089] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.615108] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.615122] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.615136] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.615149] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.615159] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.615173] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302033615028, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.615187] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.615195] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.615262] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.615275] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.615284] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.615296] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.616121] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.617555] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.617580] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.617588] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.617596] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.617605] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:33.617625] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=18] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-05-15 09:40:33.617644] WDIAG [SQL.PC] run_plan_cache_task (ob_plan_cache.cpp:2766) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] fail to update plan cache memory sys val(ret=-4029) [2025-05-15 09:40:33.617656] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=7] start lib cache evict(tenant_id=1002, mem_hold=2097152, mem_limit=1943688310, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:33.617672] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=2097152, mem_limit=1943688310, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:33.617682] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=8] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:33.620144] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=6] Dumping All Cache Objs(alloc_obj_list.count()=0, alloc_obj_list=[]) [2025-05-15 09:40:33.620162] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=16] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:33.620186] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.620212] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.620231] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.624794] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12335896147, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.624819] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.624863] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.624883] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.624897] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.624915] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.624927] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.624934] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033624773) [2025-05-15 09:40:33.625010] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:33.625028] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302033625008, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302033625008, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:33.625067] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.625098] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033625061) [2025-05-15 09:40:33.625114] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302033424957, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.625142] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12335724713, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.625158] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.625176] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.625191] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.625200] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.625215] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.625227] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.625235] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033625130) [2025-05-15 09:40:33.626250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.634854] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=6) [2025-05-15 09:40:33.636364] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.646475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.651061] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4050][LSUserTblUp6][T0][YB427F000001-000635291D6D140F-0-0] [lt=25][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001, task={tenant_id:1002, ls_id:{id:1}, add_timestamp:1747300526212290}) [2025-05-15 09:40:33.656598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.662152] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.662181] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10609558, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.662202] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10609558, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.662228] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:33.662245] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=10609558, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:33.662262] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:33.662282] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:33.662301] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.662320] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.662334] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.662352] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.662375] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033662372, replica_locations:[]}) [2025-05-15 09:40:33.662397] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.662422] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.662436] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:33.663916] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.663940] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.663956] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.663980] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.663997] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.664012] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.664029] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.664043] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033664043, replica_locations:[]}) [2025-05-15 09:40:33.664117] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10607565, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=240, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.666711] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.671006] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:371, tid:4304}]) [2025-05-15 09:40:33.676820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.683651] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=84] tx gc loop thread is running(MTL_ID()=1001) [2025-05-15 09:40:33.683685] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=32] try gc retain ctx [2025-05-15 09:40:33.683692] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-05-15 09:40:33.683702] INFO [STORAGE.TRANS] refresh_elr_tenant_config_ (ob_tx_elr_util.cpp:42) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] refresh tenant config success(tenant_id=1001, *this={last_refresh_ts:1747302033683256, can_tenant_elr:true}) [2025-05-15 09:40:33.683740] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=17] finish update min start scn(max_decided_scn={val:1747289697900328000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289695031638001, v:0}, keep_alive_scn_:{val:1747289695031638001, v:0}, update_ts_:1747302033683256}) [2025-05-15 09:40:33.684321] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=21] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:33.686234] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4491][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=28] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1aa57f82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:33.688453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.698574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.702288] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.703126] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.703153] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033703120) [2025-05-15 09:40:33.703171] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302033503062, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.703208] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12335951105, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.703230] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.703259] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.703279] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.703291] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.703310] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12335951105, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.703325] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.703337] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.703350] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.703357] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.703373] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.703391] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.703398] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] [COMPACTION DAG STATUS] (start_time=1747302023702570, end_time=1747302033703396, dag_status={}) [2025-05-15 09:40:33.703409] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033703187) [2025-05-15 09:40:33.703427] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703446] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703459] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703473] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703482] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=9] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703505] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703513] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703526] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703533] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:33.703549] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-15 09:40:33.703577] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [4453][T1001_DagSchedu][T1001][Y0-0000000000000000-0-0] [lt=26] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-15 09:40:33.708684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.715174] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.715196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.715218] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.715308] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.715344] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=34][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.715358] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.715367] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.715381] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302033715141, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.715393] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.715405] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.715482] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.715505] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.715514] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.715529] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.718797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.719733] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44868141875, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:33.720257] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.720278] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.720300] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.721392] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:33.721416] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.721428] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.721446] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:33.721454] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:33.721467] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:33.724840] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.724875] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033724834) [2025-05-15 09:40:33.724895] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302033524770, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.724933] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12335996700, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.724955] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.724983] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.725002] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.725016] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.725033] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.725046] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.725054] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033724910) [2025-05-15 09:40:33.725080] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12335824207, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.725102] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.725123] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.725140] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.725152] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.725170] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.725183] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.725189] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033725066) [2025-05-15 09:40:33.728906] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.732132] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:33.732152] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.732170] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.739017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.741437] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3953][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:33.749126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.753988] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.754007] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:33.754017] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:33.754035] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:33.754052] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=12478464) [2025-05-15 09:40:33.754080] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=28, size_used=0, mem_used=0) [2025-05-15 09:40:33.754117] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=33] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:33.759234] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.764246] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.764272] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10507451, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.764287] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10507451, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.764312] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.764329] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.764338] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.764355] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.764372] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033764370, replica_locations:[]}) [2025-05-15 09:40:33.764392] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.764413] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:240, local_retry_times:240, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:33.764439] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.764458] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:33.764471] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:33.764478] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:33.764513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:33.765622] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.765650] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.765684] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.765701] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.765717] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.765740] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.765755] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.765769] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.765785] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.765800] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033765799, replica_locations:[]}) [2025-05-15 09:40:33.765818] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.765833] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.765847] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.765865] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:33.765874] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:33.765886] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:33.765906] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:33.765922] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:33.765931] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:33.765944] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:33.765952] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:33.765964] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:33.765974] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:33.765987] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:33.765995] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:33.766009] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:33.766020] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:33.766030] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:33.766043] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:33.766055] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:33.766068] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.766077] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:33.766090] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:33.766105] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:33.766112] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=241, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:33.766139] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] will sleep(sleep_us=100000, remain_us=10505542, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=241, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.769347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.779462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.789578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.799685] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.802341] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.803199] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12336051620, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.803223] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.803246] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.803263] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.803272] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.803289] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12336051620, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.803304] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.803320] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.803334] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.803341] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.803355] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.803368] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.803376] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033803180) [2025-05-15 09:40:33.804412] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4486][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=17] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:33.809792] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.815167] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.815188] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.815208] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.815228] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.815243] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.815257] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.815266] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.815279] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302033815145, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.815294] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.815301] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.815369] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.815381] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.815390] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:33.815403] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:33.819826] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=14] gc stale ls task succ [2025-05-15 09:40:33.819900] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.820293] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14] LogLoopThread round_cost_time(us)(round_cost_time=102) [2025-05-15 09:40:33.820319] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.820337] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.820350] WDIAG [PALF] get_leader_from_cache_ (log_sliding_window.cpp:2149) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock_get_leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:33.820363] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_fetch_log_dst failed or invalid(ret=-4721, palf_id=1, self="127.0.0.1:2882", fetch_log_dst="0.0.0.0:0") [2025-05-15 09:40:33.820373] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.821453] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:33.821476] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.821496] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.821534] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:33.821546] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:33.821555] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:33.824907] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12336096152, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.824930] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.824955] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.824973] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.824982] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.824998] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.825009] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:33.825020] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033824893) [2025-05-15 09:40:33.825136] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.825162] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033825131) [2025-05-15 09:40:33.825179] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302033625130, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.825208] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12335924716, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.825224] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.825245] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:33.825260] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.825269] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.825285] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.825293] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:33.825304] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033825198) [2025-05-15 09:40:33.830076] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=78][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.832191] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:33.832212] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.832223] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.834947] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:33.840193] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.850308] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.856508] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=20] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:33.860435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.863219] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4463][T1001_TimerWK3_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=13] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:33.866257] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.866281] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10405439, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.866298] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10405439, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.866324] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.866340] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.866349] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.866366] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.866383] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033866381, replica_locations:[]}) [2025-05-15 09:40:33.866402] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.866422] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:241, local_retry_times:241, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:33.866445] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.866464] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:33.866476] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:33.866482] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:33.866529] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:33.867649] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.867676] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.867706] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.867722] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.867738] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.867761] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.867776] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.867790] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.867806] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.867818] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033867818, replica_locations:[]}) [2025-05-15 09:40:33.867836] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.867846] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.867860] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.867878] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:33.867891] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:33.867904] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:33.867923] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:33.867939] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:33.867947] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:33.867959] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:33.867967] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:33.867979] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:33.867989] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:33.868002] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:33.868009] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:33.868018] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:33.868031] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:33.868040] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:33.868052] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:33.868065] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:33.868079] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.868092] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:33.868104] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:33.868120] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:33.868127] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=242, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:33.868150] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=10403531, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=242, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.870561] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.875020] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1747302033875018, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-05-15 09:40:33.880677] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.880905] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1747302033880902, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-05-15 09:40:33.880943] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.880967] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.881010] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=42][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:33.881033] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:33.890788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.900900] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.902398] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:33.903244] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.903279] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033903237) [2025-05-15 09:40:33.903296] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302033703186, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.903336] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12336152133, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.903414] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=75] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.903437] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.903451] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.903464] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:33.903482] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12336152133, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:33.903508] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.903524] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:33.903537] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:33.903546] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:33.903560] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.903572] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:33.903579] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302033903319) [2025-05-15 09:40:33.911013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.912968] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:33.912973] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=35] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:33.913126] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:33.913147] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.913163] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:33.913188] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:33.915200] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.915219] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:33.915237] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:33.915256] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.915271] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:33.915285] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.915298] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:33.915308] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:33.915320] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302033915179, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.915335] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:33.915342] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:33.915406] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:33.915418] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:33.919981] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:33.920379] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.920400] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.920417] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.921126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.921512] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:33.921535] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.921547] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.921565] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:33.921577] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:33.921591] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:33.924958] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:33.924987] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302033924951) [2025-05-15 09:40:33.925005] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302033724909, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:33.925037] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12336196676, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:33.925057] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.925084] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:33.925101] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:33.925114] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.925204] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12336024183, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:33.925240] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:33.925259] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.931238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.932254] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:33.932284] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.932295] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:33.933742] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4822167985, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:33.933773] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.933789] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1ab65efee0) [2025-05-15 09:40:33.933804] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4822167985, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:33.941354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.951466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.957928] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:33.958511] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:33.958539] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:33.958553] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:33.958572] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:33.958591] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11796848934, flush_io_finished_ret_:0, flush_io_finished_round_:1508, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:33.961596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.968270] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.968294] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10303426, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.968312] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10303426, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.968327] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.968345] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.968361] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.968375] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.968391] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.968408] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033968407, replica_locations:[]}) [2025-05-15 09:40:33.968427] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:33.968448] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:242, local_retry_times:242, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:33.968471] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:33.968517] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:33.968530] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:33.968537] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:33.968557] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:33.969671] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.969699] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.969732] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:33.969748] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.969765] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:33.969787] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.969803] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:33.969812] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:33.969827] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:33.969842] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302033969841, replica_locations:[]}) [2025-05-15 09:40:33.969860] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.969875] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:33.969890] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:33.969908] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:33.969918] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:33.969930] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:33.969949] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:33.969966] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:33.969974] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:33.969987] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:33.969994] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:33.970006] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:33.970016] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:33.970029] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:33.970041] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:33.970050] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:33.970061] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:33.970069] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:33.970081] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:33.970094] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:33.970111] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:33.970129] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:33.970142] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:33.970157] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:33.970168] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=243, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:33.970193] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] will sleep(sleep_us=100000, remain_us=10301488, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=243, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:33.971624] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:42, tid:3961}]) [2025-05-15 09:40:33.971711] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.981820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.991936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:33.995533] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDA-0-0] [lt=12][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:33.995566] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDA-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.996846] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDB-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:33.998183] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.998212] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:33.998221] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.998235] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:33.998243] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:33.998258] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:33.998271] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:33.998309] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=35] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:33.998342] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:33.998344] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:33.998364] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:33.998375] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:33.998391] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:33.998406] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:33.998419] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:33.998429] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:33.998441] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:33.998501] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=15] gc stale ls task succ [2025-05-15 09:40:34.002048] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.002454] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.003319] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12336251656, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.003347] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.003374] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.003400] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1001}, delta=12336251656, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.003415] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.007677] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=21] ====== tenant freeze timer task ====== [2025-05-15 09:40:34.012164] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.015306] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.020437] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=34][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.020457] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.020468] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.021576] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:34.021598] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.021610] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.021628] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:34.021640] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:34.021649] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:34.022110] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:34.022147] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:34.022278] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.025017] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:34.025054] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=12336296195, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.025075] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.025252] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.025278] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034025247) [2025-05-15 09:40:34.025297] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302033825194, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.025321] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12336124767, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.025338] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.032308] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:34.032328] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.032338] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.032397] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.035042] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:34.042508] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.052620] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.058621] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.058644] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.058668] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:34.058683] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.058685] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC9686-0-0] [lt=25][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302034058654}) [2025-05-15 09:40:34.058699] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:34.058722] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:34.058737] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:34.058754] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:34.059100] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.059114] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:34.059129] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:34.059149] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=9160, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:34.062731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.065768] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4577][T1002_TimerWK0_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=5] ====== check clog disk timer task ====== [2025-05-15 09:40:34.065793] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4577][T1002_TimerWK0_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:34.067870] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221552919, support_ssl=false) [2025-05-15 09:40:34.067996] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221552919, sess.client_addr_="172.22.0.1:45714") [2025-05-15 09:40:34.068017] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552919, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.068241] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.068281] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.068308] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.068328] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.068339] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.068351] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.068380] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.068402] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552919, proxy_sessid=0) [2025-05-15 09:40:34.068443] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=14][errcode=0] server close connection(sessid=3221552919, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.068456] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552919, proxy_sessid=0) [2025-05-15 09:40:34.068464] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:34.068480] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906B-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552919, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.068544] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552919, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034067453, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.068569] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552919, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.068584] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552919, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034067453, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.068603] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552919) [2025-05-15 09:40:34.068624] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552919, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.069031] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:34.069055] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.069072] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:34.070331] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.070354] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10201372, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.070372] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10201372, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.070389] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.070404] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=10201372, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.070421] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.070441] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.070458] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.070475] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.070500] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.070513] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.070533] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034070530, replica_locations:[]}) [2025-05-15 09:40:34.070553] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.070580] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.070594] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.070642] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.070697] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11797961172, cache_obj->added_lc()=false, cache_obj->get_object_id()=15766, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.071841] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:32, tid:4630}]) [2025-05-15 09:40:34.072014] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.072035] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.072051] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.072066] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.072074] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.072089] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.072101] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.072117] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.072133] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.072147] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.072164] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.072175] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034072175, replica_locations:[]}) [2025-05-15 09:40:34.072245] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=10199437, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=244, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.072842] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.082958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.093068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.102525] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.103186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.103308] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] ====== tenant freeze timer task ====== [2025-05-15 09:40:34.103361] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.103386] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034103354) [2025-05-15 09:40:34.103403] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302033903310, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.103440] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12336351194, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.103460] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.103496] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.103515] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.103527] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.103546] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12336352253, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.103562] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.103577] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.103591] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.103598] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.103609] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.103621] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.103630] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034103420) [2025-05-15 09:40:34.108037] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552920, support_ssl=false) [2025-05-15 09:40:34.108111] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221552920, sess.client_addr_="172.22.0.1:45728") [2025-05-15 09:40:34.108129] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de16db0, session_id:3221552920, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108161] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552921, support_ssl=false) [2025-05-15 09:40:34.108190] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221552921, sess.client_addr_="172.22.0.1:45730") [2025-05-15 09:40:34.108207] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552921, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108231] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552922, support_ssl=false) [2025-05-15 09:40:34.108258] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221552922, sess.client_addr_="172.22.0.1:45740") [2025-05-15 09:40:34.108243] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221552923, support_ssl=false) [2025-05-15 09:40:34.108269] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552922, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108305] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] sm conn init succ(conn.sessid_=3221552923, sess.client_addr_="172.22.0.1:45744") [2025-05-15 09:40:34.108324] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de2c030, session_id:3221552923, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:149}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108372] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.108412] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.108434] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.108454] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.108444] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.108471] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.108505] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.108478] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.108482] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=37][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.108521] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=38][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.108521] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=43][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.108523] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=33][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.108530] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.108536] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.108539] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.108540] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.108547] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.108550] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.108557] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.108555] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108567] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108572] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552920, proxy_sessid=0) [2025-05-15 09:40:34.108572] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108586] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.108585] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552921, proxy_sessid=0) [2025-05-15 09:40:34.108588] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552922, proxy_sessid=0) [2025-05-15 09:40:34.108603] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=13][errcode=0] server close connection(sessid=3221552920, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108609] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221552924, support_ssl=false) [2025-05-15 09:40:34.108619] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=10][errcode=0] server close connection(sessid=3221552922, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108617] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=13][errcode=0] server close connection(sessid=3221552921, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108628] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552921, proxy_sessid=0) [2025-05-15 09:40:34.108628] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552922, proxy_sessid=0) [2025-05-15 09:40:34.108636] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:34.108635] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:34.108619] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552920, proxy_sessid=0) [2025-05-15 09:40:34.108647] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=27][errcode=-4016] session is null [2025-05-15 09:40:34.108645] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909C-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552921, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.108644] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906C-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552922, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.108652] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552924, sess.client_addr_="172.22.0.1:45756") [2025-05-15 09:40:34.108662] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC905F-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552920, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.108681] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.108690] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=8][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.108697] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.108691] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=37] accept one succ(*s={this:0x7f1a1de16030, session_id:3221552924, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108697] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552921, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108710] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.108722] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108724] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552922, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108735] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552923, proxy_sessid=0) [2025-05-15 09:40:34.108751] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=7][errcode=0] server close connection(sessid=3221552923, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.108747] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de16db0, session_id:3221552920, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108764] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552923, proxy_sessid=0) [2025-05-15 09:40:34.108767] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552921, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.108773] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:34.108777] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221552922, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.108785] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] kill and revert session(conn.sessid_=3221552920, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.108779] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906D-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552923, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.108792] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552921, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.108805] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221552921) [2025-05-15 09:40:34.108823] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552921, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.108854] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552922, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.108868] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552922) [2025-05-15 09:40:34.108855] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2c030, session_id:3221552923, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:149}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.108884] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552923, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.108880] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552922, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.108896] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a1de16db0, session_id:3221552920, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.108897] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de2c030, session_id:3221552923, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:149}, err:5, last_decode_time:1747302034107700, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.108908] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] mark session id unused(conn.sessid_=3221552920) [2025-05-15 09:40:34.108914] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552923) [2025-05-15 09:40:34.108920] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552920, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.108937] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552923, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.108983] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.109015] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.109030] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.109047] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.109064] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.109071] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221552925, support_ssl=false) [2025-05-15 09:40:34.109073] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.109092] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109107] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552924, proxy_sessid=0) [2025-05-15 09:40:34.109108] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552925, sess.client_addr_="172.22.0.1:45758") [2025-05-15 09:40:34.109127] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=9][errcode=0] server close connection(sessid=3221552924, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109141] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552924, proxy_sessid=0) [2025-05-15 09:40:34.109148] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:34.109122] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552925, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.109155] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909D-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552924, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.109194] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221552926, support_ssl=false) [2025-05-15 09:40:34.109210] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de16030, session_id:3221552924, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034108759, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.109249] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=37] kill and revert session(conn.sessid_=3221552924, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.109257] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de16030, session_id:3221552924, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034108759, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.109255] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221552926, sess.client_addr_="172.22.0.1:45770") [2025-05-15 09:40:34.109275] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552924) [2025-05-15 09:40:34.109278] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552926, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.109297] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552924, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.109407] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.109438] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.109462] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.109481] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.109496] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.109504] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.109519] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109525] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.109535] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552925, proxy_sessid=0) [2025-05-15 09:40:34.109555] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=9][errcode=0] server close connection(sessid=3221552925, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109555] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.109566] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.109570] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552925, proxy_sessid=0) [2025-05-15 09:40:34.109580] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:34.109579] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.109583] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221552927, support_ssl=false) [2025-05-15 09:40:34.109587] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.109594] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.109586] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9060-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552925, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.109614] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109629] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552926, proxy_sessid=0) [2025-05-15 09:40:34.109647] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=8][errcode=0] server close connection(sessid=3221552926, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109643] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552925, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302034108759, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.109649] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221552927, sess.client_addr_="172.22.0.1:45776") [2025-05-15 09:40:34.109660] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552926, proxy_sessid=0) [2025-05-15 09:40:34.109664] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552925, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.109667] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:34.109665] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552927, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.109673] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552925, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302034108759, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.109686] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221552925) [2025-05-15 09:40:34.109678] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906E-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552926, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.109701] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552925, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.109718] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552926, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302034108759, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.109740] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552926, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.109754] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552926, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302034108759, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.109775] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552926) [2025-05-15 09:40:34.109787] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552926, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.109868] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.109894] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.109907] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.109922] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.109930] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.109941] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.109954] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109967] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552927, proxy_sessid=0) [2025-05-15 09:40:34.109985] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=7][errcode=0] server close connection(sessid=3221552927, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.109996] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552927, proxy_sessid=0) [2025-05-15 09:40:34.110003] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:34.110014] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909E-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552927, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.110056] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552927, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034109820, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.110078] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552927, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.110085] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552927, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034109820, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.110102] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552927) [2025-05-15 09:40:34.110112] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552927, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.110452] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552928, support_ssl=false) [2025-05-15 09:40:34.110502] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552928, sess.client_addr_="172.22.0.1:45780") [2025-05-15 09:40:34.110517] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221552928, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.110664] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.110692] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.110715] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.110741] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.110750] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.110764] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.110779] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.110795] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552928, proxy_sessid=0) [2025-05-15 09:40:34.110815] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=8][errcode=0] server close connection(sessid=3221552928, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.110833] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552928, proxy_sessid=0) [2025-05-15 09:40:34.110854] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=20][errcode=-4016] session is null [2025-05-15 09:40:34.110868] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9061-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552928, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.110906] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552929, support_ssl=false) [2025-05-15 09:40:34.110931] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221552928, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034109820, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.110946] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552929, sess.client_addr_="172.22.0.1:45796") [2025-05-15 09:40:34.110954] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552928, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.110959] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552929, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.110962] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221552928, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034109820, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.110978] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552928) [2025-05-15 09:40:34.110989] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221552928, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.111022] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552930, support_ssl=false) [2025-05-15 09:40:34.111057] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552930, sess.client_addr_="172.22.0.1:45800") [2025-05-15 09:40:34.111072] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552930, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.111138] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.111163] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.111174] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.111189] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.111203] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.111209] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.111230] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.111236] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.111239] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552929, proxy_sessid=0) [2025-05-15 09:40:34.111265] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=15][errcode=0] server close connection(sessid=3221552929, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.111262] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.111276] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552929, proxy_sessid=0) [2025-05-15 09:40:34.111283] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:34.111277] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.111291] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC906F-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552929, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.111294] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.111316] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=22][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.111323] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.111341] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.111335] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552929, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034110879, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.111355] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552930, proxy_sessid=0) [2025-05-15 09:40:34.111355] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552929, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.111363] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552929, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034110879, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.111379] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221552929) [2025-05-15 09:40:34.111379] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=7][errcode=0] server close connection(sessid=3221552930, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.111388] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552930, proxy_sessid=0) [2025-05-15 09:40:34.111394] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:34.111391] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552929, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.111404] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC909F-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552930, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.111456] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552930, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034110879, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.111477] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552930, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.111485] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552930, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034110879, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.111512] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] mark session id unused(conn.sessid_=3221552930) [2025-05-15 09:40:34.111524] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552930, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.113306] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.114991] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:34.115011] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:34.115019] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=8] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:34.115037] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:34.115397] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.120052] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=17] gc stale ls task succ [2025-05-15 09:40:34.120251] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:34.120517] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.120541] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.120559] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.121641] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.123402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.125078] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.125106] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034125071) [2025-05-15 09:40:34.125125] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302033925020, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.125162] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12336396803, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.125182] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.125209] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.125229] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.125244] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.125262] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.125277] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.125290] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034125141) [2025-05-15 09:40:34.125320] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12336224310, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.125341] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.125361] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.125377] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.125389] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.125400] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.125412] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.125424] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034125306) [2025-05-15 09:40:34.128003] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:34.128030] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:34.128045] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:34.128064] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:34.128083] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11797018423, flush_io_finished_ret_:0, flush_io_finished_round_:1508, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:34.132365] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.133519] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.143636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.153757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.163873] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.172055] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:60, tid:4602}]) [2025-05-15 09:40:34.172407] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.172436] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10099304, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.172457] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=10099304, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.172473] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.172500] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=10099304, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.172518] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.172539] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.172558] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.172576] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.172589] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.172609] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.172631] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034172628, replica_locations:[]}) [2025-05-15 09:40:34.172653] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.172684] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.172701] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.172741] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.172823] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798063291, cache_obj->added_lc()=false, cache_obj->get_object_id()=15767, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.173988] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.174245] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.174266] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.174284] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.174299] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.174313] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.174328] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.174345] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.174356] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.174372] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.174386] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.174399] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.174415] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034174414, replica_locations:[]}) [2025-05-15 09:40:34.174457] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=24] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.174498] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=10097184, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=245, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.174537] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=31] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.174829] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.174887] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.184106] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.184548] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:34.184592] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=40][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:34.184621] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=27][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:34.184637] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:34.184649] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:34.184662] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:34.184678] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:34.184691] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:34.184704] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:34.184713] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:34.184725] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:34.184738] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:34.184752] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:34.184762] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:34.184786] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:34.184800] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:34.184815] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=14][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:34.184829] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:34.184842] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:34.184858] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:34.184865] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:34.184888] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:34.184910] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:34.184922] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:34.184928] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:34.184949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:34.184965] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6456-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.184978] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:34.184992] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.185001] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:34.185013] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:34.185026] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302034184259, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.185042] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:34.185050] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.185133] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:34.185148] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:34.185162] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=13][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:34.185175] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=13][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:34.185188] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:34.185202] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:34.185214] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6456-0-0] [lt=12][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:34.191362] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:34.194223] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.202605] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.203595] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.203629] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034203587) [2025-05-15 09:40:34.203647] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302034103419, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.203687] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12336451782, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.203708] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.203754] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.203772] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.203782] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.203801] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12336451782, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.203816] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.203832] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.203846] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.203854] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.203869] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.203882] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.203889] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034203663) [2025-05-15 09:40:34.204340] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.214463] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.215360] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.215383] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.215401] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.215417] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.215428] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.215440] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.215451] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.215461] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.215472] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302034215340, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.215485] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.215503] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.215562] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.215578] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.215586] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.215593] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.220574] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.220598] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.220719] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=120][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.224591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.225148] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12336496331, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.225172] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.225194] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.225208] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.225219] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.225232] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.225241] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.225248] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034225132) [2025-05-15 09:40:34.225372] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.225407] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034225367) [2025-05-15 09:40:34.225421] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302034025311, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.225443] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12336324895, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.225458] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.225475] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.225495] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.225504] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.225516] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.225525] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.225533] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034225432) [2025-05-15 09:40:34.235074] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.235132] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:34.245838] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.254181] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.254205] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:34.254222] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:34.254237] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:34.254250] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-15 09:40:34.254287] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=23][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:34.254290] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:34.254305] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.254322] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.254341] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.254355] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.254367] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.254381] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.254399] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.254417] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.254431] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.254442] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.254458] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.254498] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.254512] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=12] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:34.255958] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.256015] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:34.256041] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:34.256055] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:34.256075] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=18][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:34.256089] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:34.256099] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:34.256112] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:34.256119] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:34.256129] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:34.256138] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:34.256147] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:34.256157] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:34.256169] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:34.256177] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:34.256196] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:34.256206] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:34.256217] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:34.256228] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:34.256240] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:34.256253] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:34.256261] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:34.256280] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:34.256298] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:34.256308] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:34.256317] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:34.256332] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:34.256344] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.256354] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:34.256364] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.256374] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:34.256383] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=8][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:34.256393] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302034255804, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.256405] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:34.256414] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.256495] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:34.256507] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=11][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:34.256518] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:34.256527] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=9][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:34.256535] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=6][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:34.256545] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:34.256556] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BC-0-0] [lt=10][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:34.258533] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=36] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:34.258574] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] temporary file module use no blocks [2025-05-15 09:40:34.258588] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:34.258606] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:34.258654] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=47] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11797148952, flush_io_finished_ret_:0, flush_io_finished_round_:1508, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:34.266072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.266465] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=30] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:34.274639] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.274665] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9997068, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.274752] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=88][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9997068, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.274769] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.274784] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9997068, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.274798] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.274816] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.274834] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.274850] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.274862] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.274879] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.274900] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034274897, replica_locations:[]}) [2025-05-15 09:40:34.274921] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.274949] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.274961] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.275001] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.275062] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798165538, cache_obj->added_lc()=false, cache_obj->get_object_id()=15768, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.276181] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.276439] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.276460] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.276476] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.276502] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.276521] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.276536] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.276552] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.276567] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.276581] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.276593] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.276608] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.276619] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034276618, replica_locations:[]}) [2025-05-15 09:40:34.276688] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=9994994, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=246, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.286290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.296421] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.297178] WDIAG load_file_to_string (utility.h:673) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:34.297201] WDIAG get_ethernet_speed (utility.cpp:575) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:34.297211] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:34.297220] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:34.302642] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.303655] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12336552295, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.303687] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.303710] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.303727] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.303738] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.303752] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12336552295, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.303774] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.303786] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.303797] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.303807] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.303819] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.303829] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.303838] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034303633) [2025-05-15 09:40:34.306573] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.315430] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.315450] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.315468] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.315498] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.315511] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.315529] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.315544] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.315560] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.315577] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302034315410, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.315593] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.315606] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.315669] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.315683] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.315696] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.315710] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.316681] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.320527] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:34.320626] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.320649] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.320664] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.325196] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:34.325215] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.325254] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034325191) [2025-05-15 09:40:34.325269] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302034125140, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.325303] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12336596820, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.325323] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.325351] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.325366] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=4812, clean_start_pos=12400000, clean_num=200000) [2025-05-15 09:40:34.325371] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.325384] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.325404] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.325415] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.325425] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034325286) [2025-05-15 09:40:34.325443] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12336424327, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.325465] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.325500] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.325514] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.325526] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.325539] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.325548] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.325556] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034325431) [2025-05-15 09:40:34.326794] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.336905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.347017] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.353871] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-05-15 09:40:34.353904] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.353918] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:34.353929] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:34.353938] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:34.354015] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=77][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:34.354027] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.354035] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:34.354044] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:34.354053] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:34.354109] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=56][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:34.357126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.360323] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:34.360399] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=24] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:34.360469] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:34.360553] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:34.360585] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:4667, total:4677) [2025-05-15 09:40:34.360623] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:34.360672] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=43] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:4667, total:4677) [2025-05-15 09:40:34.360695] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:34.360715] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:34.360730] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:34.360749] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628769][2nd=1896]}) [2025-05-15 09:40:34.367236] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.376811] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.376836] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9894886, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.376912] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=77][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9894886, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.376926] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.376935] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9894886, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.376949] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.376962] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.376976] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.376991] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.377003] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.377019] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.377035] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034377033, replica_locations:[]}) [2025-05-15 09:40:34.377052] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.377077] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.377088] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.377116] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.377164] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798267641, cache_obj->added_lc()=false, cache_obj->get_object_id()=15769, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.377346] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.378269] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.378290] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.378306] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.378320] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.378332] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.378346] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.378361] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.378375] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.378388] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.378399] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.378416] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.378430] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034378430, replica_locations:[]}) [2025-05-15 09:40:34.378499] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=9893182, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=247, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.387455] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.387819] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.387819] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4291][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=9] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}) [2025-05-15 09:40:34.395891] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.396111] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=17] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.396157] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.396235] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.396344] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.396360] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.396614] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:34.397576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.402697] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.403697] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.403734] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034403690) [2025-05-15 09:40:34.403749] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302034203662, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.403775] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12336651780, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.403791] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.403878] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.403893] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.403903] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.403918] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12336651780, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.403930] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.403945] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.403956] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.403965] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.403978] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.403987] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.403993] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034403762) [2025-05-15 09:40:34.407684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.413281] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.415577] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.415597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.415619] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.415634] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.415645] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.415657] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.415669] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.415679] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.415691] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302034415551, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.415704] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.415714] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.415787] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.415798] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.415807] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.415817] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.418822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.420679] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.420700] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.420715] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.425262] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12336697332, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.425283] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.425303] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.425319] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.425329] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.425342] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.425350] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.425358] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034425249) [2025-05-15 09:40:34.425497] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.425519] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034425483) [2025-05-15 09:40:34.425532] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302034225432, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.425555] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12336524839, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.425570] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.425586] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.425600] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.425610] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.425622] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.425630] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.425638] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034425545) [2025-05-15 09:40:34.428937] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.435226] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:34.439050] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.449163] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.459273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.469390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.478625] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.478648] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9793073, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.478665] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9793073, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.478679] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.478688] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9793073, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.478702] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.478719] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.478734] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.478748] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.478760] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.478772] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.478788] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034478787, replica_locations:[]}) [2025-05-15 09:40:34.478806] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.478830] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.478841] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.478875] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.478921] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798369398, cache_obj->added_lc()=false, cache_obj->get_object_id()=15770, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.479499] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.480004] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.480024] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.480040] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.480055] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.480062] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.480072] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.480087] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.480101] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.480115] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.480126] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.480141] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.480155] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034480154, replica_locations:[]}) [2025-05-15 09:40:34.480216] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=9791465, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=248, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.489612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.498401] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.498424] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.498434] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:34.498445] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:34.498453] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:34.498465] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:34.498481] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:34.498504] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:34.498545] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.498538] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:34.499721] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.502755] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.503766] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12336752262, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.503792] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.503817] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.503836] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.503850] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.503872] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12336752262, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.503888] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.503909] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.503925] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.503938] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.503955] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.503967] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.503980] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034503749) [2025-05-15 09:40:34.509841] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.515582] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.515601] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.515620] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.515635] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.515646] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.515658] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.515667] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.515676] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.515686] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302034515561, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.515698] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.515709] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.515772] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.515783] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.515792] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.515803] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.519952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.520739] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.520761] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.520776] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.525315] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.525339] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034525309) [2025-05-15 09:40:34.525350] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302034325285, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.525377] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12336796801, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.525392] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.525409] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.525423] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.525432] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.525446] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.525455] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.525462] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034525366) [2025-05-15 09:40:34.525560] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12336624308, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.525578] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.525596] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.525606] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.525617] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.525631] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.525640] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.525646] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034525544) [2025-05-15 09:40:34.525667] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=16] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:34.529037] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=31] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:34.529076] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:34.530074] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.540186] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.540277] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:34.540306] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4579][T1002_TimerWK2_TenantMetaMemMg][T1002][Y0-0000000000000000-0-0] [lt=40] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:728, free_obj_hold_cnt:0, allocator used:2795520, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=3002880, tablets_mem_limit=7774753240, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:9, free_obj_hold_cnt:0, allocator used:18432, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=728) [2025-05-15 09:40:34.541915] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:34.550296] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.559607] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:34.559634] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:34.559655] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:34.559690] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:34.559704] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:34.560051] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.560067] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:34.560090] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:34.560404] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.568721] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6676-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.568793] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6676-0-0] [lt=71][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.568813] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6676-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.568831] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6676-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.568842] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.568854] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.568865] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.568875] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.568886] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302034568698, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.568898] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.568908] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=8][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:34.568976] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=13][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:34.568991] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=14][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:34.569002] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=11][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:34.569012] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=9][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:34.569024] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=9][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:34.569036] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=9][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:34.569045] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6676-0-0] [lt=8][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:34.569791] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221552939, support_ssl=false) [2025-05-15 09:40:34.569879] INFO pn_ratelimit (group.c:653) [3955][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=11] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-15 09:40:34.569924] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552939, sess.client_addr_="172.22.0.1:45816") [2025-05-15 09:40:34.569941] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552939, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.570197] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.570234] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.570245] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:281) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=10] mysql login queue(mysql_queue->queue_.size()=1) [2025-05-15 09:40:34.570270] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=32][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.570309] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=37][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.570327] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.570335] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.570369] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=20] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.570382] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552939, proxy_sessid=0) [2025-05-15 09:40:34.570414] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=13][errcode=0] server close connection(sessid=3221552939, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.570426] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552939, proxy_sessid=0) [2025-05-15 09:40:34.570437] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:34.570448] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9062-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552939, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.570512] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.570513] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552939, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034570160, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.570537] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552939, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.570547] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552939, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034570160, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.570566] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552939) [2025-05-15 09:40:34.570581] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552939, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.580341] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.580369] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9691355, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.580386] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9691355, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.580402] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.580411] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9691355, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.580428] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.580448] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.580468] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.580485] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.580508] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.580522] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.580541] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034580540, replica_locations:[]}) [2025-05-15 09:40:34.580562] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.580590] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.580605] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.580645] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.580694] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798471170, cache_obj->added_lc()=false, cache_obj->get_object_id()=15771, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.581828] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.581852] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.581871] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.581888] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.581896] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.581913] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.581932] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.581949] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.581966] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.581975] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.581992] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.582004] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034582003, replica_locations:[]}) [2025-05-15 09:40:34.582069] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=9689613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=249, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.582830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.592952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.601102] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.601126] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.601137] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to get percent [2025-05-15 09:40:34.601146] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3956][TimerWK3_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to calculate global bound size(ret=-5627, ids[i]=1001) [2025-05-15 09:40:34.601172] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.601252] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=81][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.601261] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:34.601274] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:34.601281] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:34.601294] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] failed to get percent [2025-05-15 09:40:34.601301] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3956][TimerWK3_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] failed to calculate global bound size(ret=-5157, ids[i]=1002) [2025-05-15 09:40:34.602820] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.603067] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.603813] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.603841] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034603805) [2025-05-15 09:40:34.603856] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302034403758, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.603893] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12336851791, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.603915] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.603941] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.603961] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.603971] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.603991] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12336851791, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.604007] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.604025] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.604035] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.604048] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.604059] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.604068] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.604082] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034603873) [2025-05-15 09:40:34.610121] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] new mysql sessid created(conn.sessid_=3221552940, support_ssl=false) [2025-05-15 09:40:34.610189] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552940, sess.client_addr_="172.22.0.1:45828") [2025-05-15 09:40:34.610204] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552941, support_ssl=false) [2025-05-15 09:40:34.610214] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552940, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610238] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552942, support_ssl=false) [2025-05-15 09:40:34.610252] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552943, support_ssl=false) [2025-05-15 09:40:34.610277] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552941, sess.client_addr_="172.22.0.1:45842") [2025-05-15 09:40:34.610293] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552941, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610310] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=42] sm conn init succ(conn.sessid_=3221552942, sess.client_addr_="172.22.0.1:45836") [2025-05-15 09:40:34.610326] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221552942, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610341] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] sm conn init succ(conn.sessid_=3221552943, sess.client_addr_="172.22.0.1:45846") [2025-05-15 09:40:34.610361] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552943, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610442] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.610479] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.610480] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.610516] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.610521] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.610536] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.610540] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.610539] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.610550] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.610558] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.610554] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.610559] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.610567] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.610567] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.610591] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610594] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610590] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.610609] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552941, proxy_sessid=0) [2025-05-15 09:40:34.610609] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552940, proxy_sessid=0) [2025-05-15 09:40:34.610618] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=26][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.610626] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.610638] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.610620] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552944, support_ssl=false) [2025-05-15 09:40:34.610645] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.610648] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=9][errcode=0] server close connection(sessid=3221552941, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610660] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552941, proxy_sessid=0) [2025-05-15 09:40:34.610661] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610667] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:34.610664] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.610670] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=11][errcode=0] server close connection(sessid=3221552940, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610676] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552942, proxy_sessid=0) [2025-05-15 09:40:34.610684] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221552944, sess.client_addr_="172.22.0.1:45856") [2025-05-15 09:40:34.610680] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A0-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552941, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.610688] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552940, proxy_sessid=0) [2025-05-15 09:40:34.610705] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:34.610704] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=14][errcode=0] server close connection(sessid=3221552942, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610716] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552942, proxy_sessid=0) [2025-05-15 09:40:34.610724] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:34.610712] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552944, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610726] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.610715] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9070-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552940, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.610731] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9063-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552942, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.610741] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.610736] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552941, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610753] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.610762] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552941, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.610764] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.610771] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552940, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610780] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610777] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552941, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.610794] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552943, proxy_sessid=0) [2025-05-15 09:40:34.610799] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552941) [2025-05-15 09:40:34.610802] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221552942, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610823] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552940, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.610820] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=13][errcode=0] server close connection(sessid=3221552943, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.610832] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221552942, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.610818] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552941, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.610831] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552943, proxy_sessid=0) [2025-05-15 09:40:34.610843] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:34.610840] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552940, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.610853] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552940) [2025-05-15 09:40:34.610852] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A1-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552943, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.610867] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552940, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.610886] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221552942, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.610903] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552942) [2025-05-15 09:40:34.610904] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552943, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610916] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552942, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.610938] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221552943, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.610951] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221552945, support_ssl=false) [2025-05-15 09:40:34.610948] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552943, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.610967] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552943) [2025-05-15 09:40:34.610976] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.610988] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221552945, sess.client_addr_="172.22.0.1:45866") [2025-05-15 09:40:34.611000] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552945, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.610992] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552943, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.611028] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.611036] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221552946, support_ssl=false) [2025-05-15 09:40:34.611045] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.611070] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.611073] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221552946, sess.client_addr_="172.22.0.1:45858") [2025-05-15 09:40:34.611086] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.611088] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552946, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.611097] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.611122] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611137] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552944, proxy_sessid=0) [2025-05-15 09:40:34.611163] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=13][errcode=0] server close connection(sessid=3221552944, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611174] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552944, proxy_sessid=0) [2025-05-15 09:40:34.611186] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:34.611200] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9071-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552944, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.611210] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.611233] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.611230] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552947, support_ssl=false) [2025-05-15 09:40:34.611240] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.611257] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.611272] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.611272] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552947, sess.client_addr_="172.22.0.1:45880") [2025-05-15 09:40:34.611264] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552944, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.611287] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552944, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.611283] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.611286] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552947, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.611295] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552944, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.611309] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221552944) [2025-05-15 09:40:34.611304] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611320] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552945, proxy_sessid=0) [2025-05-15 09:40:34.611327] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.611327] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221552944, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.611346] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=10][errcode=0] server close connection(sessid=3221552945, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611358] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.611359] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552945, proxy_sessid=0) [2025-05-15 09:40:34.611374] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:34.611375] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.611384] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=8][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.611392] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.611384] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9072-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552945, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.611401] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.611413] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611425] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552946, proxy_sessid=0) [2025-05-15 09:40:34.611429] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552945, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.611444] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=9][errcode=0] server close connection(sessid=3221552946, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611448] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552945, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.611452] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=7][errcode=-4018] get session fail(ret=-4018, sessid=3221552946, proxy_sessid=0) [2025-05-15 09:40:34.611459] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:34.611458] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552945, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.611470] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221552945) [2025-05-15 09:40:34.611465] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A2-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552946, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.611483] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552945, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.611498] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.611521] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.611538] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.611560] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.611553] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552946, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.611569] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.611573] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221552946, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.611581] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.611584] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552946, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034610398, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.611598] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611614] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552947, proxy_sessid=0) [2025-05-15 09:40:34.611634] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=8][errcode=0] server close connection(sessid=3221552947, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.611648] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552947, proxy_sessid=0) [2025-05-15 09:40:34.611657] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:34.611669] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9064-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552947, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.611600] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552946) [2025-05-15 09:40:34.611753] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=125] connection close(sessid=3221552946, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.611849] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=57][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552947, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.611881] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221552947, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.611932] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552947, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.611956] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] mark session id unused(conn.sessid_=3221552947) [2025-05-15 09:40:34.611974] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221552947, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.612054] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552948, support_ssl=false) [2025-05-15 09:40:34.612090] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221552948, sess.client_addr_="172.22.0.1:45894") [2025-05-15 09:40:34.612107] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552948, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.612234] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.612266] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.612266] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552949, support_ssl=false) [2025-05-15 09:40:34.612283] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.612297] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552950, support_ssl=false) [2025-05-15 09:40:34.612300] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.612307] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221552949, sess.client_addr_="172.22.0.1:45904") [2025-05-15 09:40:34.612313] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.612318] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552949, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.612325] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.612338] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221552950, sess.client_addr_="172.22.0.1:45912") [2025-05-15 09:40:34.612343] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.612371] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=26][errcode=-4018] get session fail(ret=-4018, sessid=3221552948, proxy_sessid=0) [2025-05-15 09:40:34.612378] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552950, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.612396] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=13][errcode=0] server close connection(sessid=3221552948, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.612404] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552948, proxy_sessid=0) [2025-05-15 09:40:34.612415] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:34.612422] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9073-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552948, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.612462] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552948, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.612485] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552948, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.612498] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=52] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.612507] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552948, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.612513] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.612523] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552948) [2025-05-15 09:40:34.612524] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.612536] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.612537] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552948, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.612546] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.612551] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.612559] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.612565] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.612580] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.612577] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.612593] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552950, proxy_sessid=0) [2025-05-15 09:40:34.612603] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.612612] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=9][errcode=0] server close connection(sessid=3221552950, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.612614] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.612621] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552950, proxy_sessid=0) [2025-05-15 09:40:34.612624] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.612628] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:34.612638] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.612636] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A3-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552950, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.612655] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552949, proxy_sessid=0) [2025-05-15 09:40:34.612681] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=14][errcode=0] server close connection(sessid=3221552949, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.612676] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552950, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.612696] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552949, proxy_sessid=0) [2025-05-15 09:40:34.612700] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552950, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.612704] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:34.612710] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552950, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.612713] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9065-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552949, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.612730] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221552950) [2025-05-15 09:40:34.612743] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552950, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.612769] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552949, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.612794] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552949, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.612809] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552949, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302034611457, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.612830] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552949) [2025-05-15 09:40:34.612852] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552949, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.613185] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.615709] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.615727] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.615744] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.615760] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.615775] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.615792] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.615803] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.615813] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.615824] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302034615687, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.615837] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.615848] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.615911] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.615922] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.615932] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.615941] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.620804] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.620828] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.620841] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.623301] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.625376] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.625406] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034625371) [2025-05-15 09:40:34.625424] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302034525362, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.625454] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12336897399, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.625471] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.625504] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.625522] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.625532] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.625546] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.625555] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.625563] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034625439) [2025-05-15 09:40:34.625606] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.625626] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034625602) [2025-05-15 09:40:34.625639] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302034425545, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.625660] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12336724906, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.625674] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.625690] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.625703] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.625714] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.625726] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.625734] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.625742] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034625650) [2025-05-15 09:40:34.633415] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.635317] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=17] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-05-15 09:40:34.643533] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.653151] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EADCE-0-0] [lt=25][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302034254484}) [2025-05-15 09:40:34.653642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.663750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.673840] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4336][T1_ArcSender][T1][YB427F000001-000635291B9C6112-0-0] [lt=25][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:34.673860] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.682228] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.682263] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9589479, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.682285] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9589479, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.682300] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.682314] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9589479, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.682330] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.682350] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.682370] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.682385] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.682397] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.682415] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.682435] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034682433, replica_locations:[]}) [2025-05-15 09:40:34.682455] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.682486] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.682508] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.682545] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.682609] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798573082, cache_obj->added_lc()=false, cache_obj->get_object_id()=15772, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.683966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.684001] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.684024] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.684040] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.684052] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.684063] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.684077] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.684093] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.684107] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.684121] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.684141] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.684157] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.684172] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034684171, replica_locations:[]}) [2025-05-15 09:40:34.684243] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=9587439, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=250, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.695602] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.702875] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.703882] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12336952380, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.703910] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.704035] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=124][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.704058] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.704073] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.704119] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=39] get wrs ts(ls_id={id:1001}, delta=12336952380, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.704137] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.704156] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.704171] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.704184] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.704202] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.704214] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.704226] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034703865) [2025-05-15 09:40:34.705711] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.715699] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.715724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.715744] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.715760] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.715773] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.715788] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.715800] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.715810] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.715820] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302034715675, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.715820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.715834] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.715844] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.715910] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.715923] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.715933] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.715940] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.720858] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.720878] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.720892] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.725442] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=12336996918, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.725465] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.725485] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.725510] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.725520] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.725534] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.725542] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.725550] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034725428) [2025-05-15 09:40:34.725666] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.725705] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034725660) [2025-05-15 09:40:34.725721] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302034625650, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.725743] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12336824425, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.725766] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.725781] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.725794] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.725803] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.725816] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.725824] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.725834] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034725733) [2025-05-15 09:40:34.725930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.725931] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:34.730277] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [4579][T1002_TimerWK2_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=51] get_replay_process success(id={id:1}, submitted_log_size=61925697, unsubmitted_log_size=61925697, replayed_log_size=61925697, unreplayed_log_size=0) [2025-05-15 09:40:34.730312] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [4579][T1002_TimerWK2_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=31] get_replay_process success(id={id:1001}, submitted_log_size=9195983, unsubmitted_log_size=9195983, replayed_log_size=9195983, unreplayed_log_size=0) [2025-05-15 09:40:34.730339] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [4579][T1002_TimerWK2_ReplayProcessSt][T1002][Y0-0000000000000000-0-0] [lt=10] dump tenant replay process(tenant_id=1002, unsubmitted_log_size(MB)=67, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=67, replayed_log_size(MB)=67, last_submitted_log_size(MB)=67, last_replayed_log_size(MB)=67, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-15 09:40:34.736039] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.741526] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3954][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=25] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:34.746153] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.754334] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.754353] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:34.754364] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:34.754380] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:34.754411] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=28] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:34.754445] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=45] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:34.756265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.766377] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.773232] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:334, tid:4304}]) [2025-05-15 09:40:34.776496] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.784365] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.784394] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9487333, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.784411] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9487333, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.784425] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.784433] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9487333, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.784447] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.784463] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.784478] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.784504] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.784513] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.784528] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.784545] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034784544, replica_locations:[]}) [2025-05-15 09:40:34.784563] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.784583] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:250, local_retry_times:250, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:34.784605] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.784626] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:34.784637] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:34.784646] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:34.784681] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:34.785821] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.785848] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.785880] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.785896] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.785911] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.785933] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.785947] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.785959] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.785970] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.785985] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034785984, replica_locations:[]}) [2025-05-15 09:40:34.786001] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.786015] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.786028] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.786045] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:34.786056] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:34.786068] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:34.786085] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:34.786100] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:34.786111] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:34.786122] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:34.786134] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:34.786142] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:34.786155] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:34.786166] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:34.786177] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:34.786186] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:34.786197] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:34.786207] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:34.786219] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:34.786235] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:34.786246] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:34.786258] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:34.786270] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:34.786283] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:34.786294] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=251, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:34.786319] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=9485363, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=251, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.786609] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.787320] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_TTLTabletMgr][T1002][YB427F000001-000635291F9C6660-0-0] [lt=21][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.787345] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_TTLTabletMgr][T1002][YB427F000001-000635291F9C6660-0-0] [lt=25][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.787355] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4337) [4579][T1002_TimerWK2_TTLTabletMgr][T1002][YB427F000001-000635291F9C6660-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:34.787370] INFO [SERVER] run_task (ob_plugin_vector_index_scheduler.cpp:1077) [4579][T1002_TimerWK2_TTLTabletMgr][T1002][YB427F000001-000635291F9C6660-0-0] [lt=13] schema service not ready(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.796723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.802930] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.803929] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.803958] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034803923) [2025-05-15 09:40:34.803979] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302034603872, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.804011] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12337051917, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.804030] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.804054] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.804075] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.804090] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.804113] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12337051917, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.804129] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.804147] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.804163] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.804176] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.804192] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.804204] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.804216] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034803996) [2025-05-15 09:40:34.806813] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.815769] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.815792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.815835] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=35][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.815856] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.815868] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.815880] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.815891] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.815901] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302034815745, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.815915] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.815926] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.815992] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.816005] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.816015] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.816025] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.816928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.819886] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=17] gc stale ls task succ [2025-05-15 09:40:34.820920] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.820942] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.820954] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.822055] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:34.822075] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.822086] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.822103] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:34.822115] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:34.822127] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:34.825504] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.825535] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034825500) [2025-05-15 09:40:34.825552] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302034625438, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.825575] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12337097508, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.825589] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.825609] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.825626] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.825636] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.825659] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.825669] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.825678] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034825564) [2025-05-15 09:40:34.825734] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12336925015, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.825756] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.825773] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.825786] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.825795] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.825808] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.825816] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.825824] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034825723) [2025-05-15 09:40:34.827041] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.827215] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4463][T1001_TimerWK3_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=18] ====== check clog disk timer task ====== [2025-05-15 09:40:34.827232] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4463][T1001_TimerWK3_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=14] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:34.832771] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:34.832790] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.832832] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.835408] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-05-15 09:40:34.837159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.847272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.857077] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=29] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:34.857387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.857655] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4243][T1_Occam][T1][YB427F000001-000635291B4C65CF-0-0] [lt=15] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.867503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.877616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.882637] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=28] PNIO [ratelimit] time: 1747302034882634, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:34.886442] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.886464] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9385256, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.886477] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9385256, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.886512] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.886527] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.886540] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.886555] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.886571] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034886570, replica_locations:[]}) [2025-05-15 09:40:34.886588] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.886608] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:251, local_retry_times:251, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:34.886630] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.886649] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:34.886659] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:34.886666] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:34.886686] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:34.887731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.887860] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.887888] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.887922] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.887938] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.887953] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.887974] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.887989] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.888001] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.888015] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.888029] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034888029, replica_locations:[]}) [2025-05-15 09:40:34.888046] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.888060] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.888070] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.888086] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:34.888097] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:34.888108] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:34.888125] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:34.888140] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:34.888153] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:34.888161] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:34.888171] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:34.888182] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:34.888195] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:34.888207] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:34.888217] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:34.888224] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:34.888235] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:34.888245] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:34.888253] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:34.888268] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:34.888281] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:34.888294] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:34.888305] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:34.888317] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:34.888326] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=252, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:34.888351] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=9383330, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=252, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.888524] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=22] PNIO [ratelimit] time: 1747302034888518, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:34.892362] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.892385] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.892398] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:34.892421] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:34.897344] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] ====== tenant freeze timer task ====== [2025-05-15 09:40:34.897840] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.902986] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:34.904007] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12337152477, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.904032] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.904134] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=101][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.904157] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.904170] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.904191] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12337152477, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:34.904207] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.904225] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:34.904241] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:34.904254] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:34.904268] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.904282] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:34.904295] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034903991) [2025-05-15 09:40:34.907949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.913378] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:34.913399] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.913410] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:34.913428] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:34.915122] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=40] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:34.915120] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:34.915807] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.915828] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.915849] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.915864] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.915876] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.915889] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.915903] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.915913] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.915923] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302034915783, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.915937] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.915947] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:34.916014] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:34.916027] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:34.916035] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:34.916042] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:34.918057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.920977] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.920997] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.921011] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.922113] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:34.922132] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.922147] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.922162] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:34.922173] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:34.922185] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:34.925574] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12337197008, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:34.925597] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.925617] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:34.925631] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.925640] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.925653] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.925662] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:34.925672] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034925562) [2025-05-15 09:40:34.925784] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:34.925809] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302034925778) [2025-05-15 09:40:34.925823] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302034725733, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:34.925847] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12337024515, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:34.925862] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:34.925880] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:34.925892] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:34.925901] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:34.925914] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.925923] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:34.925935] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302034925837) [2025-05-15 09:40:34.926196] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:34.928170] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.932831] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:34.932851] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.932864] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:34.934141] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4825207114, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:34.934243] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=29] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:34.934260] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4825207114, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:34.935394] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.935415] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.935431] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.935446] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:34.935466] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.935482] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.935500] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.935513] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.935525] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.935540] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:34.935552] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:34.935564] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:34.935585] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.935597] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.935611] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.935630] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.935642] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.935654] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.935666] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.935678] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034935678, replica_locations:[]}) [2025-05-15 09:40:34.935696] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.935709] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.935725] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.935746] WDIAG [SERVER] refresh_sys_tenant_ls (ob_service.cpp:2711) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=20][errcode=-4721] fail to refresh sys tenant log stream(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, SYS_LS={id:1}) [2025-05-15 09:40:34.935758] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:34.935805] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:34.935827] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.935862] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=35] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:34.935883] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:34.935906] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=23] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=120, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:34.935929] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:34.935953] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:34.935963] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:34.938285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.948399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.958510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.958524] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=28] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:34.961096] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4531][T1002_Occam][T1002][YB427F000001-000635291EDC65C9-0-0] [lt=24] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.968618] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.973647] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:28, tid:3961}]) [2025-05-15 09:40:34.978724] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.988466] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.988512] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9283227, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.988611] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=101][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9283227, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.988622] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.988630] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9283227, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.988658] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.988675] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.988690] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.988703] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.988715] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.988730] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.988744] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034988744, replica_locations:[]}) [2025-05-15 09:40:34.988761] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:34.988781] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:252, local_retry_times:252, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:34.988801] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:34.988813] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:34.988829] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:34.988831] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:34.988840] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:34.988849] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:34.988886] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:34.988900] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.988956] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798879436, cache_obj->added_lc()=false, cache_obj->get_object_id()=15775, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.990079] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.990107] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.990139] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:34.990153] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.990168] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:34.990181] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:34.990189] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:34.990202] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:34.990218] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:34.990231] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.990244] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:34.990257] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:34.990272] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:34.990285] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302034990284, replica_locations:[]}) [2025-05-15 09:40:34.990301] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.990314] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:34.990326] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:34.990342] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:34.990349] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:34.990360] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:34.990378] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:34.990393] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:34.990403] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:34.990415] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:34.990423] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:34.990433] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:34.990447] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:34.990459] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:34.990469] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:34.990480] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:34.990499] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:34.990507] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:34.990519] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:34.990533] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:34.990544] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:34.990556] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:34.990568] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:34.990581] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:34.990591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=253, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:34.990615] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=9281066, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=253, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:34.995597] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDC-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.995615] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDC-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.995630] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDC-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.995645] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDC-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.995657] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CDC-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.995671] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:34.995683] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.995695] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.995704] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302034995567, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:34.995717] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.995727] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:34.995799] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:34.997071] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDD-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.997088] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDD-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:34.997099] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDD-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:34.997112] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:34.997123] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CDD-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:34.997136] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:34.997148] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:34.997155] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:34.997165] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302034997061, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:34.997177] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:34.997184] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:34.997237] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:34.997628] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552957, support_ssl=false) [2025-05-15 09:40:34.997740] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552957, sess.client_addr_="127.0.0.1:45114") [2025-05-15 09:40:34.997761] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552957, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.998000] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:34.998039] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:34.998067] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:34.998086] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:34.998099] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:34.998110] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:34.998137] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.998156] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552957, proxy_sessid=0) [2025-05-15 09:40:34.998196] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=12][errcode=0] server close connection(sessid=3221552957, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:34.998207] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552957, proxy_sessid=0) [2025-05-15 09:40:34.998216] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:34.998228] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9074-0-0] [lt=9] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552957, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=3908101, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:34.998272] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552957, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034997756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:118, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:34.998293] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552957, proxy_sessid=0, ret=0) [2025-05-15 09:40:34.998303] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552957, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302034997756, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:118, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:34.998319] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552957) [2025-05-15 09:40:34.998334] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552957, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:34.998562] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=15] gc stale ls task succ [2025-05-15 09:40:34.998596] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.998606] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:34.998622] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:34.998624] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:34.998631] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:34.998636] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:34.998638] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:34.998646] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:34.998647] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:34.998660] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:34.998674] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:34.998686] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:34.998705] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=59][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:34.998712] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:34.998721] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:34.998719] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:34.998732] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:34.998941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.003043] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.004051] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.004087] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035004044) [2025-05-15 09:40:35.004101] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302034803994, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.004131] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12337252900, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.004147] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.004167] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.004182] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.004192] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.004207] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12337252900, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.004219] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.004234] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.004243] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.004250] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.004261] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.004271] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.004279] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035004114) [2025-05-15 09:40:35.009052] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.015921] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.015943] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.015966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.015982] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.015995] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.016007] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.016021] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.016029] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.016040] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302035015895, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.016053] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.016063] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.016135] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.016148] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.016158] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.016166] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.019164] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.021042] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.021064] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.021080] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:35.022174] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:35.022194] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.022206] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:35.022222] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:35.022233] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:35.022245] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:35.025624] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.025647] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035025620) [2025-05-15 09:40:35.025661] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302034825563, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.025679] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:35.025705] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12337297295, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.025718] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.025735] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.025751] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.025760] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.025846] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.025882] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035025840) [2025-05-15 09:40:35.025910] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302034925837, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.025935] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12337124802, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.025949] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.025965] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.029279] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.032888] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:35.032908] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.032940] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=31][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:35.033640] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:35.033676] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:35.038522] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] Cache replace map node details(ret=0, replace_node_count=0, replace_time=3013, replace_start_pos=5900000, replace_num=100000) [2025-05-15 09:40:35.038551] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=10) [2025-05-15 09:40:35.039394] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.049509] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.054904] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4599][T1002_CdcSrv][T1001][YB427F000001-000635291D3C61A9-0-0] [lt=6][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.054925] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4599][T1002_CdcSrv][T1001][YB427F000001-000635291D3C61A9-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.054942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4599][T1002_CdcSrv][T1001][YB427F000001-000635291D3C61A9-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.054957] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4599][T1002_CdcSrv][T1001][YB427F000001-000635291D3C61A9-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.054967] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4599][T1002_CdcSrv][T1002][YB427F000001-000635291D3C61A9-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.054979] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc) [2025-05-15 09:40:35.054991] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.055001] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.055010] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302035054884, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc) [2025-05-15 09:40:35.055023] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.055032] WDIAG [SHARE] get_valid_dest_pairs (ob_archive_persist_helper.cpp:455) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] failed to exec sql(ret=-5627, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='path' order by dest_no asc) [2025-05-15 09:40:35.055089] WDIAG [CLOG.EXTLOG] query_tenant_archive_info_ (ob_cdc_service.cpp:648) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] get_valid_dest_pairs failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:35.055101] WDIAG [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:198) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] query_tenant_archive_info_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:35.055113] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=11] finish to resize log external storage handler(current_ts=1747302035055111, tenant_max_cpu=13, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-15 09:40:35.055126] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [4599][T1002_CdcSrv][T1002][Y0-0000000000000000-0-0] [lt=11] snapshot_traffic_info once [2025-05-15 09:40:35.059623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.060531] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.060560] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.060587] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:35.060599] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:35.060612] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.060605] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC968D-0-0] [lt=26][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302035060573}) [2025-05-15 09:40:35.060627] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:35.060638] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.060655] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:35.060665] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:35.060675] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:35.060682] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:35.060691] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:35.060700] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:35.060712] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.060722] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.061062] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:35.061076] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.061091] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:35.061100] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:35.061109] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:35.061117] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:35.061127] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:35.061136] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:35.061151] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.061166] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.069158] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=13] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:35.069191] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.069202] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:35.069213] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:35.069223] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:35.069305] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:35.069738] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.071308] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552960, support_ssl=false) [2025-05-15 09:40:35.071405] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221552960, sess.client_addr_="172.22.0.1:45916") [2025-05-15 09:40:35.071426] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221552960, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.071648] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.071693] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.071719] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.071734] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.071745] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.071756] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.071781] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.071797] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552960, proxy_sessid=0) [2025-05-15 09:40:35.071833] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=11][errcode=0] server close connection(sessid=3221552960, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.071843] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552960, proxy_sessid=0) [2025-05-15 09:40:35.071852] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:35.071864] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A4-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552960, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.071916] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221552960, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035070589, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.071942] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221552960, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.071956] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221552960, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035070589, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.071978] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552960) [2025-05-15 09:40:35.072003] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221552960, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.079855] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.089972] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.090749] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.090776] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9180952, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.090794] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9180952, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.090809] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.090827] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9180952, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.090848] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.090872] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.090889] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.090901] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.090913] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.090929] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.090946] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035090944, replica_locations:[]}) [2025-05-15 09:40:35.090964] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.090984] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:253, local_retry_times:253, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:35.091008] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.091019] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:35.091035] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:35.091046] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:35.091052] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:35.091070] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:35.091083] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.091141] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11798981612, cache_obj->added_lc()=false, cache_obj->get_object_id()=15776, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.092446] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:35.092475] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.092520] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.092533] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.092545] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.092555] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.092563] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.092573] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.092589] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.092604] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.092617] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.092629] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.092640] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.092653] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035092653, replica_locations:[]}) [2025-05-15 09:40:35.092671] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:35.092685] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.092697] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:35.092713] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:35.092721] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:35.092732] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:35.092751] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:35.092766] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:35.092773] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:35.092785] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:35.092792] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:35.092799] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:35.092809] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:35.092823] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:35.092831] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:35.092841] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:35.092852] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:35.092861] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:35.092872] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:35.092885] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:35.092897] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.092907] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:35.092918] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:35.092931] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:35.092941] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=254, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:35.092966] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=9178716, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=254, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.100086] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.103105] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.104128] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12337352250, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.104156] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.104179] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.104205] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12337352250, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.104218] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.110199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.112129] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552962, support_ssl=false) [2025-05-15 09:40:35.112129] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] new mysql sessid created(conn.sessid_=3221552961, support_ssl=false) [2025-05-15 09:40:35.112196] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221552962, sess.client_addr_="172.22.0.1:45924") [2025-05-15 09:40:35.112196] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552961, sess.client_addr_="172.22.0.1:45922") [2025-05-15 09:40:35.112211] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de14030, session_id:3221552962, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112215] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112259] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552963, support_ssl=false) [2025-05-15 09:40:35.112291] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=73] new mysql sessid created(conn.sessid_=3221552964, support_ssl=false) [2025-05-15 09:40:35.112318] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] sm conn init succ(conn.sessid_=3221552964, sess.client_addr_="172.22.0.1:45954") [2025-05-15 09:40:35.112314] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552963, sess.client_addr_="172.22.0.1:45938") [2025-05-15 09:40:35.112331] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221552964, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112331] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221552963, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112353] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221552965, support_ssl=false) [2025-05-15 09:40:35.112399] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552965, sess.client_addr_="172.22.0.1:45968") [2025-05-15 09:40:35.112415] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de20030, session_id:3221552965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:149}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112426] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.112466] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.112470] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.112505] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.112522] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.112529] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.112552] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.112551] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.112570] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.112577] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.112585] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.112595] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.112609] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.112614] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112558] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.112623] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.112639] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=22][errcode=-4018] get session fail(ret=-4018, sessid=3221552962, proxy_sessid=0) [2025-05-15 09:40:35.112643] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112639] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=78][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.112656] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552963, proxy_sessid=0) [2025-05-15 09:40:35.112663] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=23][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.112678] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=15][errcode=0] server close connection(sessid=3221552962, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112679] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=11][errcode=0] server close connection(sessid=3221552963, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112691] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552962, proxy_sessid=0) [2025-05-15 09:40:35.112691] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552963, proxy_sessid=0) [2025-05-15 09:40:35.112688] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=25][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.112701] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:35.112701] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:35.112713] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.112721] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.112712] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9075-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552962, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.112712] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A5-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552963, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.112737] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.112738] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.112728] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=17] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112749] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.112749] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.112761] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.112761] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.112756] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=23][errcode=-4018] get session fail(ret=-4018, sessid=3221552961, proxy_sessid=0) [2025-05-15 09:40:35.112768] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221552966, support_ssl=false) [2025-05-15 09:40:35.112765] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221552962, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112771] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.112771] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.112783] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552962, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.112783] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.112784] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.112771] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221552963, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112796] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221552962, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.112800] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112800] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112802] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221552966, sess.client_addr_="172.22.0.1:45974") [2025-05-15 09:40:35.112810] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552962) [2025-05-15 09:40:35.112802] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221552963, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.112811] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552964, proxy_sessid=0) [2025-05-15 09:40:35.112812] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552965, proxy_sessid=0) [2025-05-15 09:40:35.112801] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=16][errcode=0] server close connection(sessid=3221552961, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112828] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=26][errcode=-4018] get session fail(ret=-4018, sessid=3221552961, proxy_sessid=0) [2025-05-15 09:40:35.112818] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221552963, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.112842] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=9][errcode=0] server close connection(sessid=3221552965, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112848] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] mark session id unused(conn.sessid_=3221552963) [2025-05-15 09:40:35.112847] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=9][errcode=0] server close connection(sessid=3221552964, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.112848] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=18][errcode=-4016] session is null [2025-05-15 09:40:35.112856] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552965, proxy_sessid=0) [2025-05-15 09:40:35.112869] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:35.112862] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552964, proxy_sessid=0) [2025-05-15 09:40:35.112878] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=16][errcode=-4016] session is null [2025-05-15 09:40:35.112878] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A6-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552965, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.112865] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9066-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552961, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.112870] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221552963, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.112886] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9076-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552964, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.112892] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552962, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.112813] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x7f1a1de20db0, session_id:3221552966, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112928] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221552964, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112946] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552964, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.112931] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.112960] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221552961, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.112961] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=122] new mysql sessid created(conn.sessid_=3221552967, support_ssl=false) [2025-05-15 09:40:35.112967] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221552964, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.112983] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552964) [2025-05-15 09:40:35.112993] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552964, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.113000] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552961, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.113013] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221552967, sess.client_addr_="172.22.0.1:45982") [2025-05-15 09:40:35.113019] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552961) [2025-05-15 09:40:35.113035] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552967, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.113038] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552961, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.113102] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de20030, session_id:3221552965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:149}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.113132] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552965, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.113153] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f1a1de20030, session_id:3221552965, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:149}, err:5, last_decode_time:1747302035111906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.113178] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] mark session id unused(conn.sessid_=3221552965) [2025-05-15 09:40:35.113174] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221552968, support_ssl=false) [2025-05-15 09:40:35.113228] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221552968, sess.client_addr_="172.22.0.1:45986") [2025-05-15 09:40:35.113207] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552965, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.113246] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221552968, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.113287] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=41] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.113371] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=28] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.113439] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.113503] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=62][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.113519] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.113545] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=25][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.113587] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.113623] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=28][errcode=-4018] get session fail(ret=-4018, sessid=3221552966, proxy_sessid=0) [2025-05-15 09:40:35.113656] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=63] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.113671] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=14][errcode=0] server close connection(sessid=3221552966, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.113693] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=22][errcode=-4018] get session fail(ret=-4018, sessid=3221552966, proxy_sessid=0) [2025-05-15 09:40:35.113701] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:35.113720] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=40] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.113719] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.113713] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A7-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552966, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.113765] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=22] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.113756] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.113789] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=31][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.113797] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221552969, support_ssl=false) [2025-05-15 09:40:35.113802] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=37][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.113783] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de20db0, session_id:3221552966, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035112965, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.113809] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.113821] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.113825] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=38] kill and revert session(conn.sessid_=3221552966, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.113823] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.113830] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.113838] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.113833] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de20db0, session_id:3221552966, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035112965, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.113843] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.113849] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552966) [2025-05-15 09:40:35.113845] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221552969, sess.client_addr_="172.22.0.1:45998") [2025-05-15 09:40:35.113856] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552967, proxy_sessid=0) [2025-05-15 09:40:35.113852] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.113862] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552966, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.113871] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552968, proxy_sessid=0) [2025-05-15 09:40:35.113877] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=9][errcode=0] server close connection(sessid=3221552967, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.113866] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221552969, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.113888] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552967, proxy_sessid=0) [2025-05-15 09:40:35.113895] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:35.113895] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=11][errcode=0] server close connection(sessid=3221552968, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.113892] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552970, support_ssl=false) [2025-05-15 09:40:35.113908] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552968, proxy_sessid=0) [2025-05-15 09:40:35.113904] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9077-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552967, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.113924] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=16][errcode=-4016] session is null [2025-05-15 09:40:35.113934] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552970, sess.client_addr_="172.22.0.1:46014") [2025-05-15 09:40:35.113932] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9067-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552968, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.113952] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de20030, session_id:3221552970, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.113977] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221552971, support_ssl=false) [2025-05-15 09:40:35.113992] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221552968, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302035112965, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.114014] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552971, sess.client_addr_="172.22.0.1:46024") [2025-05-15 09:40:35.114024] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221552968, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.114028] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221552971, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.114044] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221552968, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302035112965, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.114070] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] mark session id unused(conn.sessid_=3221552968) [2025-05-15 09:40:35.114077] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552967, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035112965, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.114107] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221552967, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.114093] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552968, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.114131] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.114152] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552967, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035112965, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.114161] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.114172] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552967) [2025-05-15 09:40:35.114180] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.114192] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.114188] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552967, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.114205] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.114211] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.114223] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.114224] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.114237] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552969, proxy_sessid=0) [2025-05-15 09:40:35.114247] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.114254] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=7][errcode=0] server close connection(sessid=3221552969, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.114263] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552969, proxy_sessid=0) [2025-05-15 09:40:35.114270] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:35.114268] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.114268] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.114285] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.114290] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.114304] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.114308] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.114326] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.114328] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=19][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.114335] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.114346] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.114276] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A8-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552969, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.114359] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.114358] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=21] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.114375] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552971, proxy_sessid=0) [2025-05-15 09:40:35.114375] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552970, proxy_sessid=0) [2025-05-15 09:40:35.114393] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=7][errcode=0] server close connection(sessid=3221552971, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.114402] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=11][errcode=0] server close connection(sessid=3221552970, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.114412] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552971, proxy_sessid=0) [2025-05-15 09:40:35.114394] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221552969, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302035114027, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.114414] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552970, proxy_sessid=0) [2025-05-15 09:40:35.114422] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:35.114426] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:35.114425] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221552969, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.114430] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9068-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552971, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.114441] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221552969, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302035114027, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.114437] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9078-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552970, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.114467] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] mark session id unused(conn.sessid_=3221552969) [2025-05-15 09:40:35.114486] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552969, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.114520] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221552971, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302035114027, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.114563] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de20030, session_id:3221552970, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302035114027, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.114584] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552971, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.114595] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221552970, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.114610] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221552971, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302035114027, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.114632] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552971) [2025-05-15 09:40:35.114648] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552971, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.114671] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f1a1de20030, session_id:3221552970, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302035114027, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.114683] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221552970) [2025-05-15 09:40:35.114693] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552970, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.115124] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:35.115145] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:35.115154] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=8] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:35.115171] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:35.115979] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.116009] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.120111] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=15] gc stale ls task succ [2025-05-15 09:40:35.120317] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.121103] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.125690] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.125718] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035125682) [2025-05-15 09:40:35.125736] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302035025675, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.125766] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12337397766, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.125783] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.125923] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12337225273, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.125946] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.126512] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:35.130447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.140574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.150688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.160796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.164838] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=18] ====== check clog disk timer task ====== [2025-05-15 09:40:35.164874] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=31] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:35.170905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.173991] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.174065] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=24][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:31, tid:4630}]) [2025-05-15 09:40:35.174186] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.174424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.174891] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.181011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.186893] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=15] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:35.186923] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:35.186939] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:35.186953] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:35.186954] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=7] Garbage Collector is running(seq_=151, gc_interval=10000000) [2025-05-15 09:40:35.186968] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:35.186979] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:35.186995] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:35.186990] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.187006] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:35.187015] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:35.187022] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=7][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:35.187030] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:35.187039] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:35.187045] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0] gc_check_member_list_ cost time(ret=0, time_us=69) [2025-05-15 09:40:35.187049] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:35.187059] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:35.187063] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=13] execute_gc cost time(ret=0, time_us=0) [2025-05-15 09:40:35.187080] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:35.187091] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.187103] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.187114] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:35.187125] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:35.187139] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:35.187148] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:35.187168] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:35.187188] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.187197] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.187203] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:35.187219] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:35.187230] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.187241] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:35.187252] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.187263] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:35.187273] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:35.187282] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca3b0, start=1747302035186639, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.187294] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:35.187305] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.187388] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:35.187400] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=11][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:35.187411] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:35.187422] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=10][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:35.187431] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=6][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:35.187443] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:35.187452] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4460][T1001_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291CFC6457-0-0] [lt=9][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:35.188591] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4603][T1002_GCCollect][T1001][YB427F000001-00063529202C623E-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.188692] INFO [CLOG] get_ls_status_from_table (ob_garbage_collector.cpp:1481) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0] failed to get ls status info from table(ret=-5627, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:35.188708] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1666) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] get_ls_status_ failed(tmp_ret=-5627, ls->get_ls_id()={id:1}) [2025-05-15 09:40:35.189905] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4603][T1002_GCCollect][T1001][YB427F000001-00063529202C623F-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.189953] INFO [CLOG] get_ls_status_from_table (ob_garbage_collector.cpp:1481) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0] failed to get ls status info from table(ret=-5627, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:35.189968] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1666) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] get_ls_status_ failed(tmp_ret=-5627, ls->get_ls_id()={id:1001}) [2025-05-15 09:40:35.189982] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=12] execute_gc cost time(ret=0, time_us=0) [2025-05-15 09:40:35.189995] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [4603][T1002_GCCollect][T1002][Y0-0000000000000000-0-0] [lt=10] ObSafeDestroyHandler start process [2025-05-15 09:40:35.191124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.191465] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=17] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:35.193117] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.193142] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9078592, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.193161] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=9078592, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.193177] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.193187] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=9078592, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.193203] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.193221] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.193239] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.193257] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.193269] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.193286] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.193304] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035193302, replica_locations:[]}) [2025-05-15 09:40:35.193324] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.193347] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.193369] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.193381] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:35.193419] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.193471] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11799083949, cache_obj->added_lc()=false, cache_obj->get_object_id()=15777, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.194871] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.194894] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.194910] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.194924] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.194936] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.194950] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.194966] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.194982] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.194996] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.195007] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.195023] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.195038] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035195037, replica_locations:[]}) [2025-05-15 09:40:35.195110] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=9076572, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=255, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.201235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.203160] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.204168] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.204198] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035204161) [2025-05-15 09:40:35.204214] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302035004113, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.204243] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12337452452, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.204260] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.204283] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.204299] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.204310] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.204327] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12337452452, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.204339] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.204353] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.204364] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.204373] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.204384] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.204395] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.204403] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035204226) [2025-05-15 09:40:35.207892] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4462][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=13] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:35.211344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.215894] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.221453] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.222297] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.225763] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12337497885, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.225793] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.225819] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.225837] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.225851] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.225868] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.225879] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.225889] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035225742) [2025-05-15 09:40:35.225927] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.225965] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035225920) [2025-05-15 09:40:35.225981] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302035025924, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.226006] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12337325392, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.226022] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.226040] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.226054] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.226067] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.226079] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.226090] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.226096] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035225995) [2025-05-15 09:40:35.231579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.233008] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.238639] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=11] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=9) [2025-05-15 09:40:35.242062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.252741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.254501] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.254523] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:35.254534] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:35.254550] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:35.254564] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=10398720) [2025-05-15 09:40:35.254597] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=33, size_used=0, mem_used=0) [2025-05-15 09:40:35.254626] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:35.258752] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:35.258785] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-05-15 09:40:35.258797] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:35.258815] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:35.258834] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11798149175, flush_io_finished_ret_:0, flush_io_finished_round_:1509, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:35.262853] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.263184] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65CB-0-0] [lt=24] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.265729] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65CD-0-0] [lt=21] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.265959] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=21][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:35.265983] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=23][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.266004] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.266026] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.266044] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.266059] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.266077] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.266097] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.266117] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.266133] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.266149] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.266169] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.266201] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.266216] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:35.267907] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=17] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:35.267936] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:35.267950] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:35.267968] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:35.267985] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:35.267997] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:35.268016] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:35.268028] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:35.268042] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:35.268054] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:35.268067] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:35.268081] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:35.268095] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:35.268107] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:35.268131] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:35.268145] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.268160] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.268173] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:35.268187] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:35.268203] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:35.268215] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:35.268236] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:35.268260] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.268270] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.268281] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:35.268309] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=17][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:35.268323] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.268336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:35.268350] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.268363] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:35.268374] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:35.268387] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302035267648, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.268403] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:35.268415] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.268525] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=13][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:35.268541] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=15][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:35.268558] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=16][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:35.268571] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:35.268584] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:35.268599] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=14][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:35.268612] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6390-0-0] [lt=12][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:35.268758] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=36] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:35.268788] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:35.268805] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:35.268820] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:35.268832] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:35.268845] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:35.268865] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=15][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:35.268878] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=12][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:35.268889] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:35.268900] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:35.268940] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=39][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:35.268951] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:35.268963] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:35.268974] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:35.268995] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:35.269007] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.269021] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.269035] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:35.269046] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:35.269061] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:35.269072] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:35.269094] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:35.269116] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.269126] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.269135] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:35.269156] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:35.269171] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6240-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.269182] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4420][T1001_Occam][T1001][YB427F000001-000635291DAC6240-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:35.269196] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:35.269209] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:35.269221] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:35.269234] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302035268516, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:35.269246] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:35.269259] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1001, tenant_id=1, columns=0x7f1aa10563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:35.269279] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1001, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:35.269382] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1aa10563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:35.269399] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:35.269413] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:35.269431] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:35.269454] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=14] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:35.272966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.274367] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=110][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:54, tid:4602}]) [2025-05-15 09:40:35.283079] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.293194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.295238] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.295263] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8976458, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.295279] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8976458, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.295292] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.295303] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=8976458, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.295316] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.295329] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.295343] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.295356] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.295367] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.295381] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.295398] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035295395, replica_locations:[]}) [2025-05-15 09:40:35.295418] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.295441] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.295452] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:35.295474] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.295526] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11799186002, cache_obj->added_lc()=false, cache_obj->get_object_id()=15778, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.296773] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.296792] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.296807] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.296820] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.296830] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.296842] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.296856] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.296868] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.296881] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.296891] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.296904] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.296916] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035296916, replica_locations:[]}) [2025-05-15 09:40:35.296983] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8974699, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=256, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.297372] WDIAG load_file_to_string (utility.h:673) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=47][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:35.297392] WDIAG get_ethernet_speed (utility.cpp:575) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:35.297401] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:35.297415] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:35.303215] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.303302] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.304242] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12337552418, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.304270] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.304295] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.304314] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.304327] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.304344] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12337552418, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.304358] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.304372] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.304386] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.304394] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.304404] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.304415] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.304426] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035304219) [2025-05-15 09:40:35.313414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.315948] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.315969] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.315987] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.316003] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.316014] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.316027] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.316036] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.316044] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.316056] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302035315929, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.316069] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.316077] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.316130] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.316141] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.316152] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.316163] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.323526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.325802] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:35.325825] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.325874] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035325796) [2025-05-15 09:40:35.325890] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302035125747, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.325917] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12337597907, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.325937] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.325957] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.325972] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.325980] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.325996] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.326006] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.325998] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12337425414, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.326012] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035325903) [2025-05-15 09:40:35.326016] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.326037] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.326051] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.326062] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.326076] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.326086] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.326097] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035325984) [2025-05-15 09:40:35.326777] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:35.333635] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.343748] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.353855] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.354149] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:35.354183] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.354204] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:35.354217] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:35.354228] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:35.354309] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=81][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:35.354322] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.354328] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:35.354338] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:35.354348] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:35.354399] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:35.361445] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:35.361532] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:35.361606] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:35.361672] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:35.361702] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:3009, total:3021) [2025-05-15 09:40:35.361739] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:35.361787] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:3009, total:3021) [2025-05-15 09:40:35.361815] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:35.361837] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:35.361853] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:35.361874] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] successfully alloc block(macro_id={[ver=1,mode=0,seq=628770][2nd=1897]}) [2025-05-15 09:40:35.363963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.374072] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.384183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.389162] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:35.389197] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=33][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2025-05-15 09:40:35.389213] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:35.389227] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2025-05-15 09:40:35.389241] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:35.389251] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:35.389266] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2025-05-15 09:40:35.389277] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:35.389287] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:35.389296] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:35.389308] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:6094) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] resolve table failed(ret=-5019) [2025-05-15 09:40:35.389319] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:5649) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=10][errcode=-5019] resolve joined table item failed(ret=-5019) [2025-05-15 09:40:35.389329] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5457) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] resolve joined table failed(ret=-5019) [2025-05-15 09:40:35.389338] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:35.389346] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:35.389356] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:35.389365] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:35.389383] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:35.389396] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.389407] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.389417] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:35.389427] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2025-05-15 09:40:35.389439] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:35.389449] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:35.389467] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:35.389486] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.389505] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:35.389513] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:35.389531] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:35.389543] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4776][BlackListServic][T1][YB427F000001-000635291F1C6309-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.389555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4776][BlackListServic][T0][YB427F000001-000635291F1C6309-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2025-05-15 09:40:35.389567] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:35.389579] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:35.389590] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:35.389600] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302035388822, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:35.389612] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:114) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:35.389624] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:35.389719] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11] ls blacklist refresh finish(cost_time=2354) [2025-05-15 09:40:35.394297] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.395269] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.395453] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.395612] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.395647] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.395974] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=32] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.396148] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.396707] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=47] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:35.397119] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.397141] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8874582, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.397159] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8874582, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.397172] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.397200] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=8874582, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.397214] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.397230] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.397245] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.397260] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.397271] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.397283] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.397300] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035397297, replica_locations:[]}) [2025-05-15 09:40:35.397317] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.397340] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.397350] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:35.397382] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.397422] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11799287900, cache_obj->added_lc()=false, cache_obj->get_object_id()=15779, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.398626] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.398648] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.398663] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.398676] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.398687] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.398699] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.398712] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.398725] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.398740] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.398751] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.398765] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.398776] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035398776, replica_locations:[]}) [2025-05-15 09:40:35.398840] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8872842, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=257, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.400023] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.403271] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.404283] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.404315] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035404276) [2025-05-15 09:40:35.404333] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302035204225, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.404370] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12337652529, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.404389] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.404411] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.404413] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.404429] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.404442] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.404458] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12337652529, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.404472] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.404495] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.404508] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.404516] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.404530] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.404541] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.404548] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035404351) [2025-05-15 09:40:35.413515] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=51][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.414526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.416073] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.416091] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.416109] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.416138] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=27][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.416148] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.416159] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.416171] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.416183] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.416191] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302035416049, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.416205] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.416212] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.416286] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.416298] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.416309] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.416317] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.425650] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.425868] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12337697994, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.425891] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.425915] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.425931] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.425942] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.425958] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.425969] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.425980] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035425853) [2025-05-15 09:40:35.426041] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.426065] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035426036) [2025-05-15 09:40:35.426077] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302035225995, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.426102] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12337525501, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.426118] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.426135] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.426149] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.426160] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.426171] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.426182] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.426188] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035426091) [2025-05-15 09:40:35.435765] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.438719] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:35.445883] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.451173] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4634][T1002_ArcSender][T1002][YB427F000001-000635291E2C6112-0-0] [lt=21][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:35.456004] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.466116] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.476237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.486361] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.496481] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.498773] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.498775] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:35.498818] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=43][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:35.498828] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:35.498841] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:35.498850] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:35.498866] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:35.498887] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:35.498903] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:35.498944] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:35.498981] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.499004] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8772721, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.499021] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8772721, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.499036] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.499048] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=8772721, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.499062] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.499079] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.499096] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.499110] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.499121] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.499136] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.499153] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035499151, replica_locations:[]}) [2025-05-15 09:40:35.499173] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.499198] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.499208] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:35.499240] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.499295] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11799389770, cache_obj->added_lc()=false, cache_obj->get_object_id()=15780, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.500598] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.500619] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.500634] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.500647] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.500656] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.500669] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.500683] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.500696] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.500708] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.500718] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.500731] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.500741] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035500740, replica_locations:[]}) [2025-05-15 09:40:35.500807] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8770874, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=258, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.503325] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.504349] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12337752592, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.504381] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.504408] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.504430] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.504442] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.504461] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12337752592, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.504474] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.504501] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.504515] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.504522] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.504537] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.504548] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.504559] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035504312) [2025-05-15 09:40:35.506579] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.516203] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.516224] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.516245] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.516263] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.516277] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.516291] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.516304] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.516315] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.516329] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302035516169, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.516348] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.516359] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.516437] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.516449] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.516457] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.516470] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.516693] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.525920] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.525952] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035525913) [2025-05-15 09:40:35.525970] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302035325903, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.526000] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12337798114, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.526021] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.526046] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.526057] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.526068] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.526084] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.526095] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.526105] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035525987) [2025-05-15 09:40:35.526098] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.526118] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035526094) [2025-05-15 09:40:35.526133] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302035426090, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.526157] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12337625621, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.526173] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.526190] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.526205] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.526213] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.526227] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.526237] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.526243] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035526146) [2025-05-15 09:40:35.526800] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.527038] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:35.536905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.540954] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:35.541292] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:35.541325] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:35.542000] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=25] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:35.547013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.557120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.561591] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.561637] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.561654] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.561669] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:35.561691] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:35.561705] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:35.561720] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.562072] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:35.562084] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:35.562097] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:35.567230] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.570583] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6677-0-0] [lt=17][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.570605] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6677-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.570624] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6677-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.570642] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291F6C6677-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.570657] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.570671] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.570681] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.570691] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.570702] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302035570559, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.570717] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.570730] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:35.570796] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=12][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:35.570809] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=12][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:35.570822] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=12][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:35.570834] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=11][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:35.570848] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=11][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:35.570863] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=13][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:35.570873] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4578][T1002_TimerWK1_LSMetaCh][T1002][YB427F000001-000635291F6C6677-0-0] [lt=10][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:35.572594] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552983, support_ssl=false) [2025-05-15 09:40:35.572719] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552983, sess.client_addr_="172.22.0.1:46026") [2025-05-15 09:40:35.572738] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552983, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.572856] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.572892] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.572919] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=85][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.572940] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.572954] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.572967] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.572991] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.573010] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552983, proxy_sessid=0) [2025-05-15 09:40:35.573052] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=14][errcode=0] server close connection(sessid=3221552983, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.573064] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552983, proxy_sessid=0) [2025-05-15 09:40:35.573071] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:35.573085] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90A9-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552983, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.573138] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552983, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035572344, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.573161] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552983, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.573174] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552983, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035572344, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.573197] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552983) [2025-05-15 09:40:35.573214] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552983, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.577341] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.589661] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.599776] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.600937] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.600962] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8670760, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.600979] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8670760, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.600992] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.601003] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=8670760, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.601016] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.601051] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=33][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.601066] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.601080] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.601091] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.601105] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.601121] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035601119, replica_locations:[]}) [2025-05-15 09:40:35.601139] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.601161] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.601172] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:35.601208] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.601255] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11799491731, cache_obj->added_lc()=false, cache_obj->get_object_id()=15781, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.602384] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.602406] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.602421] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.602434] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:35.602441] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:35.602454] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:35.602469] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:35.602481] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.602504] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.602515] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.602528] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.602541] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035602540, replica_locations:[]}) [2025-05-15 09:40:35.602604] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8669077, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=259, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.603386] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.604374] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.604414] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035604368) [2025-05-15 09:40:35.604430] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302035404346, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.604467] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12337852925, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.604486] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.604521] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.604539] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.604551] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.604572] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12337852925, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.604593] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.604610] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.604622] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.604633] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.604647] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.604658] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.604665] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035604445) [2025-05-15 09:40:35.609895] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.614136] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552984, support_ssl=false) [2025-05-15 09:40:35.614156] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552985, support_ssl=false) [2025-05-15 09:40:35.614196] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221552984, sess.client_addr_="172.22.0.1:46040") [2025-05-15 09:40:35.614215] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604030, session_id:3221552984, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.614237] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221552985, sess.client_addr_="172.22.0.1:46052") [2025-05-15 09:40:35.614250] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221552986, support_ssl=false) [2025-05-15 09:40:35.614252] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de20db0, session_id:3221552985, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.614285] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221552986, sess.client_addr_="172.22.0.1:46066") [2025-05-15 09:40:35.614295] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552986, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.614333] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221552987, support_ssl=false) [2025-05-15 09:40:35.614362] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221552987, sess.client_addr_="172.22.0.1:46074") [2025-05-15 09:40:35.614372] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552987, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.614452] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.614464] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.614484] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.614518] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.614518] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=32][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.614539] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.614536] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.614549] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.614557] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.614556] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.614583] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.614583] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.614603] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221552985, proxy_sessid=0) [2025-05-15 09:40:35.614570] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=38][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.614605] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.614623] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.614630] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.614627] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=54][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.614639] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=12][errcode=0] server close connection(sessid=3221552985, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.614642] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.614646] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.614651] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552985, proxy_sessid=0) [2025-05-15 09:40:35.614652] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.614657] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552986, proxy_sessid=0) [2025-05-15 09:40:35.614659] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:35.614679] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=10][errcode=0] server close connection(sessid=3221552986, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.614678] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.614686] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=7][errcode=-4018] get session fail(ret=-4018, sessid=3221552986, proxy_sessid=0) [2025-05-15 09:40:35.614669] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9079-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552985, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.614693] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:35.614692] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552984, proxy_sessid=0) [2025-05-15 09:40:35.614699] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AA-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552986, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.614726] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=13][errcode=0] server close connection(sessid=3221552984, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.614724] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de20db0, session_id:3221552985, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302035613655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.614744] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552984, proxy_sessid=0) [2025-05-15 09:40:35.614746] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221552985, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.614752] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:35.614749] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552986, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302035613655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.614754] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de20db0, session_id:3221552985, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302035613655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.614774] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552985) [2025-05-15 09:40:35.614776] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221552986, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.614769] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9069-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552984, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.614791] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552986, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302035613655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.614808] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221552986) [2025-05-15 09:40:35.614789] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552985, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.614836] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.614833] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552986, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.614860] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.614875] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.614881] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221552984, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035613655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.614900] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221552984, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.614896] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.614916] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.614926] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221552984, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035613655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.614932] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.614941] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221552984) [2025-05-15 09:40:35.614954] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552984, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.614955] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.614975] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552987, proxy_sessid=0) [2025-05-15 09:40:35.614990] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221552989, support_ssl=false) [2025-05-15 09:40:35.615013] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=12][errcode=0] server close connection(sessid=3221552987, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615029] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221552987, proxy_sessid=0) [2025-05-15 09:40:35.615024] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552990, support_ssl=false) [2025-05-15 09:40:35.615044] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:35.615041] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221552989, sess.client_addr_="172.22.0.1:46078") [2025-05-15 09:40:35.615058] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907A-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552987, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.615060] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221552989, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615080] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552990, sess.client_addr_="172.22.0.1:46102") [2025-05-15 09:40:35.615095] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221552991, support_ssl=false) [2025-05-15 09:40:35.615104] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a1de20db0, session_id:3221552990, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615105] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552987, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615133] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221552987, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.615143] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221552992, support_ssl=false) [2025-05-15 09:40:35.615141] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552987, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.615141] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221552991, sess.client_addr_="172.22.0.1:46086") [2025-05-15 09:40:35.615158] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552987) [2025-05-15 09:40:35.615161] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f616030, session_id:3221552991, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615171] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552987, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.615188] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552992, sess.client_addr_="172.22.0.1:46104") [2025-05-15 09:40:35.615208] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221552992, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615271] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.615330] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=26] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.615364] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.615397] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=32][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.615402] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=37] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.615418] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.615436] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=18][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.615454] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=33] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.615457] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615477] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221552989, proxy_sessid=0) [2025-05-15 09:40:35.615486] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.615484] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.615522] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=36][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.615527] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.615531] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=32][errcode=0] server close connection(sessid=3221552989, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615549] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221552989, proxy_sessid=0) [2025-05-15 09:40:35.615544] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=22][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.615558] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:35.615548] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=43] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.615560] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=16][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.615557] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.615568] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AB-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552989, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.615577] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615578] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.615593] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.615596] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221552990, proxy_sessid=0) [2025-05-15 09:40:35.615603] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.615607] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.615616] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=9][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.615618] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.615617] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=8][errcode=0] server close connection(sessid=3221552990, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615623] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.615628] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552990, proxy_sessid=0) [2025-05-15 09:40:35.615630] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.615631] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615626] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221552989, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615636] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:35.615642] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615643] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221552992, proxy_sessid=0) [2025-05-15 09:40:35.615649] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552989, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.615654] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552991, proxy_sessid=0) [2025-05-15 09:40:35.615649] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906A-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552990, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.615663] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=10][errcode=0] server close connection(sessid=3221552992, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615660] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221552989, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.615673] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552992, proxy_sessid=0) [2025-05-15 09:40:35.615673] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=7][errcode=0] server close connection(sessid=3221552991, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.615678] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552989) [2025-05-15 09:40:35.615686] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:35.615687] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552991, proxy_sessid=0) [2025-05-15 09:40:35.615699] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:35.615702] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907B-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552992, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.615705] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AC-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552991, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.615702] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221552989, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.615713] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de20db0, session_id:3221552990, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615746] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f616030, session_id:3221552991, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615753] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221552992, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.615773] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221552991, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.615777] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221552990, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.615793] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] kill and revert session(conn.sessid_=3221552992, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.615808] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de20db0, session_id:3221552990, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.615812] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a6f616030, session_id:3221552991, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.615833] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221552991) [2025-05-15 09:40:35.615833] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] mark session id unused(conn.sessid_=3221552990) [2025-05-15 09:40:35.615850] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552991, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.615853] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221552990, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.615879] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221552992, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302035614718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.615898] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221552992) [2025-05-15 09:40:35.615913] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221552992, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.616026] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552993, support_ssl=false) [2025-05-15 09:40:35.616078] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221552993, sess.client_addr_="172.22.0.1:46108") [2025-05-15 09:40:35.616092] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de20030, session_id:3221552993, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.616111] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221552994, support_ssl=false) [2025-05-15 09:40:35.616149] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221552994, sess.client_addr_="172.22.0.1:46120") [2025-05-15 09:40:35.616165] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221552994, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.616169] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221552995, support_ssl=false) [2025-05-15 09:40:35.616219] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221552995, sess.client_addr_="172.22.0.1:46134") [2025-05-15 09:40:35.616250] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221552995, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.616331] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=36] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.616359] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.616377] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.616392] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.616393] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.616399] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.616410] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.616411] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:35.616421] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.616431] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.616437] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:35.616441] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552993, proxy_sessid=0) [2025-05-15 09:40:35.616444] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.616449] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:35.616462] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=10][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.616465] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.616474] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221552993, proxy_sessid=0) [2025-05-15 09:40:35.616475] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.616469] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:35.616481] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:35.616484] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.616485] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:35.616495] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906B-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552993, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.616505] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=19][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:35.616510] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=19] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.616519] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221552995, proxy_sessid=0) [2025-05-15 09:40:35.616513] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.616523] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:35.616532] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.616538] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=7][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.616538] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221552994, proxy_sessid=0) [2025-05-15 09:40:35.616547] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221552995, proxy_sessid=0) [2025-05-15 09:40:35.616554] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:35.616546] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de20030, session_id:3221552993, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035615777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.616554] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.616566] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221552993, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.616565] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=11][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.616562] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907C-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552995, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.616571] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.616574] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de20030, session_id:3221552993, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302035615777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.616576] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221552994, proxy_sessid=0) [2025-05-15 09:40:35.616582] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.616588] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:35.616592] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221552993) [2025-05-15 09:40:35.616592] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.616601] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.616596] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AD-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221552994, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:35.616604] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552993, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.616612] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302035616475, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.616622] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.616632] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.616636] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221552995, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035615777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.616654] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221552995, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.616647] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221552994, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302035615777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:35.616668] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221552994, proxy_sessid=0, ret=0) [2025-05-15 09:40:35.616680] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221552994, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302035615777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.616678] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221552995, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302035615777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:35.616696] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552995) [2025-05-15 09:40:35.616697] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221552994) [2025-05-15 09:40:35.616711] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.616706] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221552995, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.616711] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221552994, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:35.616721] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.616732] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.616740] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.620008] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.622571] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302035622133, register_time_us=-1, curr_time_us=1747302035622568) [2025-05-15 09:40:35.622602] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:35.622620] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", [SlidingWindow]={palf_id:1, self:"127.0.0.1:2882", lsn_allocator:{max_log_id:6194005, max_lsn:934288498, max_scn:{val:1747289697897144000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:934287681}, reuse_lsn:{lsn:934288498}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:934288498}}, last_submit_lsn:{lsn:934287681}, last_submit_end_lsn:{lsn:934288498}, last_submit_log_id:6194005, last_submit_log_pid:4, max_flushed_lsn:{lsn:934287681}, max_flushed_end_lsn:{lsn:934288498}, max_flushed_log_pid:4, committed_end_lsn:{lsn:934287681}, last_slide_log_id:6194004, last_slide_scn:{val:1747289697751194001, v:0}, last_slide_lsn:{lsn:934287559}, last_slide_end_lsn:{lsn:934287681}, last_slide_log_pid:4, last_slide_log_accum_checksum:910368143, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f1a73fca530}, [StateMgr]={this:0x7f1a73fce9b0, self:"127.0.0.1:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:-1, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7f1a73fd5470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"127.0.0.1:2882", alive_paxos_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:789328978}, prev_mode_pid:4}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:4, config_seq:5}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f1a73fcbc70}, [ModeMgr]={palf_id:1, self:"127.0.0.1:2882", applied_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:872361984}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, log_config_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:789328978}, prev_mode_pid:4}, log_snapshot_meta:{version:2, base_lsn:{lsn:872361984}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:86016}, readable_log_tail:{lsn:86016}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:939466752}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/meta", dir_fd:141, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67018752, block_header_serialize_buf_:0x7f1a73fd0bb8, flashback_version:0}, log_storage:{log_tail:{lsn:934288498}, readable_log_tail:{lsn:934288498}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:13, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:143, min_block_id:0, max_block_id:14, curr_writable_block_id:13}, logical_block_size_:67104768, curr_block_writable_size_:5178254, block_header_serialize_buf_:0x7f1a73fd28b8, flashback_version:0}, palf_epoch:2, last_purge_throttling_ts:-1, this:0x7f1a73fcf0f0}, [Reconfirm]={palf_id:1, self:"127.0.0.1:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f1a73fceb30}) [2025-05-15 09:40:35.622846] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=221][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:35.622862] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", ack_info_list=[]) [2025-05-15 09:40:35.622872] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:35.622885] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] current log_task status(palf_id=1, self="127.0.0.1:2882", log_id=6194005, log_task={header:{begin_lsn:{lsn:934287681}, end_lsn:{lsn:934288498}, log_id:6194005, min_scn:{val:1747289697897144000, v:0}, max_scn:{val:1747289697897144000, v:0}, data_len:761, proposal_id:4, prev_lsn:{lsn:934287559}, prev_proposal_id:4, committed_end_lsn:{lsn:934287681}, data_checksum:1279038126, accum_checksum:1505208473, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1747300524702173, freeze_ts:1747300524702173, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:-1747300524702174, submit_to_flush cost time:0}, ack_info_list=[]) [2025-05-15 09:40:35.622919] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:396) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4018] fail to get server region(ret=-4018, server="127.0.0.1:2882") [2025-05-15 09:40:35.622931] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4018] get_server_region failed(server="127.0.0.1:2882") [2025-05-15 09:40:35.622943] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:398) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:35.622956] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2766) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:35.622984] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] re_register_parent reason(palf_id=1001, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302035622133, register_time_us=-1, curr_time_us=1747302035622983) [2025-05-15 09:40:35.622998] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1001, self="127.0.0.1:2882") [2025-05-15 09:40:35.623010] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10] [PALF_DUMP](palf_id=1001, self="127.0.0.1:2882", [SlidingWindow]={palf_id:1001, self:"127.0.0.1:2882", lsn_allocator:{max_log_id:6139319, max_lsn:747348553, max_scn:{val:1747289697898202000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:747348431}, reuse_lsn:{lsn:747348553}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:747348553}}, last_submit_lsn:{lsn:747348431}, last_submit_end_lsn:{lsn:747348553}, last_submit_log_id:6139319, last_submit_log_pid:4, max_flushed_lsn:{lsn:747348431}, max_flushed_end_lsn:{lsn:747348553}, max_flushed_log_pid:4, committed_end_lsn:{lsn:747348431}, last_slide_log_id:6139318, last_slide_scn:{val:1747289697751194001, v:0}, last_slide_lsn:{lsn:747348309}, last_slide_end_lsn:{lsn:747348431}, last_slide_log_pid:4, last_slide_log_accum_checksum:2356512705, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f1a73f5a530}, [StateMgr]={this:0x7f1a73f5e9b0, self:"127.0.0.1:2882", palf_id:1001, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:-1, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7f1a73f65470, allow_vote:true}, [ConfigMgr]={palf_id:1001, self:"127.0.0.1:2882", alive_paxos_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:621730248}, prev_mode_pid:4}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:4, config_seq:5}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f1a73f5bc70}, [ModeMgr]={palf_id:1001, self:"127.0.0.1:2882", applied_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1746497901775524000, v:0}}, accepted_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1746497901775524000, v:0}}, last_submit_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1746497901775524000, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1001, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:738152448}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, log_config_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:621730248}, prev_mode_pid:4}, log_snapshot_meta:{version:2, base_lsn:{lsn:738152448}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1746497901775524000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1746497901775524000, v:0}}}, log_meta_storage:{log_tail:{lsn:102400}, readable_log_tail:{lsn:102400}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:805257216}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/meta", dir_fd:138, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67002368, block_header_serialize_buf_:0x7f1a73f60bb8, flashback_version:0}, log_storage:{log_tail:{lsn:747348553}, readable_log_tail:{lsn:747348553}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:11, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:140, min_block_id:0, max_block_id:12, curr_writable_block_id:11}, logical_block_size_:67104768, curr_block_writable_size_:57908663, block_header_serialize_buf_:0x7f1a73f628b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:-1, this:0x7f1a73f5f0f0}, [Reconfirm]={palf_id:1001, self:"127.0.0.1:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f1a73f5eb30}) [2025-05-15 09:40:35.623176] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=167][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1001, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:35.623186] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9] [PALF_DUMP](palf_id=1001, self="127.0.0.1:2882", ack_info_list=[]) [2025-05-15 09:40:35.623197] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1001, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:35.623208] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12] current log_task status(palf_id=1001, self="127.0.0.1:2882", log_id=6139319, log_task={header:{begin_lsn:{lsn:747348431}, end_lsn:{lsn:747348553}, log_id:6139319, min_scn:{val:1747289697898202000, v:0}, max_scn:{val:1747289697898202000, v:0}, data_len:66, proposal_id:4, prev_lsn:{lsn:747348309}, prev_proposal_id:4, committed_end_lsn:{lsn:747348431}, data_checksum:2222604892, accum_checksum:4219431924, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1747300523498417, freeze_ts:1747300523498417, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:-1747300523498418, submit_to_flush cost time:0}, ack_info_list=[]) [2025-05-15 09:40:35.623232] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:396) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4018] fail to get server region(ret=-4018, server="127.0.0.1:2882") [2025-05-15 09:40:35.623242] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] get_server_region failed(server="127.0.0.1:2882") [2025-05-15 09:40:35.623253] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:398) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1001, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:35.623264] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2766) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1001, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:35.625995] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12337897474, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.626024] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.626047] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.626065] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.626077] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.626093] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.626103] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.626112] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035625973) [2025-05-15 09:40:35.626160] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.626187] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035626154) [2025-05-15 09:40:35.626203] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302035526146, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.626227] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12337724981, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.626241] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.626258] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.626271] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.626280] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.626293] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.626301] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.626309] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035626215) [2025-05-15 09:40:35.630123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.632037] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4289][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4827309812, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:35.632422] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4289][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=23] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4827309812, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:35.634992] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=1] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-05-15 09:40:35.636369] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][YB427F000001-000635291D5C6440-0-0] [lt=27][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.636389] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4461][T1001_TimerWK1_ReqMemEvict][T1001][YB427F000001-000635291D5C6440-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.636405] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4461][T1001_TimerWK1_ReqMemEvict][T1001][YB427F000001-000635291D5C6440-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.636420] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_ReqMemEvict][T1001][YB427F000001-000635291D5C6440-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.636432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4461][T1001_TimerWK1_ReqMemEvict][T1001][YB427F000001-000635291D5C6440-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.636443] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:35.636453] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.636462] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.636473] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302035636355, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:35.636484] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.636504] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5627) [2025-05-15 09:40:35.636558] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to fetch max rule version(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.636571] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] failed to fetch max rule version(ret=-5627) [2025-05-15 09:40:35.636580] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] failed to sync rule from inner table(ret=-5627) [2025-05-15 09:40:35.638819] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:35.640235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.650351] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.653895] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4050][LSUserTblUp6][T0][YB427F000001-000635291D6D14D6-0-0] [lt=45][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001, task={tenant_id:1002, ls_id:{id:1}, add_timestamp:1747300526212290}) [2025-05-15 09:40:35.660462] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.670578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.680687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.690797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.702434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.702760] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.702787] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8568950, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.702817] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8568950, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.702849] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.702865] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.702877] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.702894] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.702913] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035702910, replica_locations:[]}) [2025-05-15 09:40:35.702932] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.702961] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.703444] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.704430] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.704453] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.704449] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12337952462, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.704469] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.704478] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.704507] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.704511] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.704521] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.704530] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.704532] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.704543] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.704548] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.704560] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12337952462, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.704563] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035704562, replica_locations:[]}) [2025-05-15 09:40:35.704574] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.704589] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.704602] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.704611] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.704626] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.704636] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.704638] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8567043, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=260, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.704646] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035704426) [2025-05-15 09:40:35.712548] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.716248] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.716271] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.716291] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.716322] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.716334] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.716346] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.716357] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.716368] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302035716228, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.716380] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.716388] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.716446] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.716457] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.716468] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.716480] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.722591] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11] LogLoopThread round_cost_time(us)(round_cost_time=756) [2025-05-15 09:40:35.722660] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.726038] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.726067] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035726031) [2025-05-15 09:40:35.726084] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302035525983, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.726107] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12337998053, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.726124] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.726142] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.726159] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.726166] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.726180] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.726188] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.726196] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035726097) [2025-05-15 09:40:35.726228] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12337825560, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.726248] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.726266] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.726279] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.726289] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.726301] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.726310] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.726318] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035726214) [2025-05-15 09:40:35.727300] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:35.732772] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.741612] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3956][TimerWK3_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:35.742886] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.752997] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.754683] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.754702] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:35.754751] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:35.754773] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:35.754788] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:35.754823] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:35.763107] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.773218] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.783333] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.793448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.803570] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.803625] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=97] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:35.803647] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:35.803657] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.804499] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.804533] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035804484) [2025-05-15 09:40:35.804641] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=108][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302035604444, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.804682] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12338052964, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.804703] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.804729] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.804748] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.804762] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.804785] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12338052964, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.804787] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.804802] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.804809] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8466916, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.804827] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8466916, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.804826] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.804846] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.804858] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.804861] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.804874] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.804882] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.804885] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.804898] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.804902] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.804912] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035804661) [2025-05-15 09:40:35.804922] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035804919, replica_locations:[]}) [2025-05-15 09:40:35.804942] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.804969] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.806408] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.806428] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.806444] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.806466] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.806480] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.806504] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.806520] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.806534] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035806533, replica_locations:[]}) [2025-05-15 09:40:35.806604] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=8465078, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=261, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.813681] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.816359] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.816383] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.816405] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.816424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.816436] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.816449] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.816460] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.816472] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302035816334, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.816486] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.816501] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.816569] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.816583] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.816592] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.816606] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.819947] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=18] gc stale ls task succ [2025-05-15 09:40:35.823761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.826105] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12338097629, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.826130] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.826152] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.826167] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.826177] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.826191] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.826201] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.826212] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035826091) [2025-05-15 09:40:35.826278] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.826314] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035826272) [2025-05-15 09:40:35.826328] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302035626215, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.826354] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12337925136, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.826368] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.826386] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.826398] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.826408] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.826420] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.826428] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.826436] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035826343) [2025-05-15 09:40:35.833870] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.838913] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=6) [2025-05-15 09:40:35.843981] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.854092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.857676] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:35.864201] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.874312] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.875596] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:366, tid:4290}]) [2025-05-15 09:40:35.884427] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.884910] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4289][T1_TimerWK0_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=18] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:35.890252] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302035890250, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:35.894574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.896133] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=31] PNIO [ratelimit] time: 1747302035896130, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0 [2025-05-15 09:40:35.903562] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:35.904540] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12338152312, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.904585] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=39] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.904683] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.904706] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=118][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.904728] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.904740] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.904756] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12338153369, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:35.904769] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.904783] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:35.904795] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:35.904804] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:35.904817] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.904828] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:35.904837] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035904507) [2025-05-15 09:40:35.905123] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.905148] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.905162] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:35.905190] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:35.906734] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.906757] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8364965, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.906776] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8364965, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.906801] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.906816] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.906828] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.906845] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.906863] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035906861, replica_locations:[]}) [2025-05-15 09:40:35.906881] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:35.906902] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:261, local_retry_times:261, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:35.906929] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:35.906947] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:35.906960] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:35.906969] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:35.906989] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:35.908169] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:35.908204] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=33][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.908237] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:35.908251] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.908267] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:35.908289] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.908303] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:35.908314] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:35.908330] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:35.908343] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302035908342, replica_locations:[]}) [2025-05-15 09:40:35.908360] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:35.908374] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.908384] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:35.908399] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:35.908410] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:35.908422] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:35.908441] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:35.908456] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:35.908466] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:35.908478] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:35.908495] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:35.908503] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:35.908517] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:35.908530] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:35.908540] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:35.908550] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:35.908560] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:35.908570] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:35.908581] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:35.908598] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:35.908610] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:35.908618] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:35.908629] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:35.908643] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:35.908654] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=262, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:35.908679] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=8363002, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=262, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:35.910273] INFO [STORAGE.TABLELOCK] garbage_collect_for_all_ls_ (ob_table_lock_service.cpp:339) [4607][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=24] finish check and clear obj lock(ls->get_ls_id()={id:1001}) [2025-05-15 09:40:35.913613] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:35.913635] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.913652] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:35.913677] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:35.914790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.916473] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.916504] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=30][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.916527] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.916546] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.916560] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.916573] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.916585] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.916595] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.916607] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302035916440, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.916622] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.916634] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:35.916719] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:35.916730] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:35.916739] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:35.916749] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:35.917505] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:35.917504] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=30] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:35.919971] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.919986] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:35.919998] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=11] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-05-15 09:40:35.920010] WDIAG [SQL.PC] run_plan_cache_task (ob_plan_cache.cpp:2766) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] fail to update plan cache memory sys val(ret=-4029) [2025-05-15 09:40:35.920024] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=9] start lib cache evict(tenant_id=1001, mem_hold=2097152, mem_limit=215965365, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:35.920035] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=11] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=2097152, mem_limit=215965365, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:35.920051] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=14] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:35.921570] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:35.921590] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.921604] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:35.922606] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=10] Dumping All Cache Objs(alloc_obj_list.count()=0, alloc_obj_list=[]) [2025-05-15 09:40:35.922626] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4462][T1001_TimerWK2_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=19] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:35.922738] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:35.922756] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.922770] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:35.922785] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:35.922797] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:35.922806] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:35.924901] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.926153] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:35.926173] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302035926150, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302035926150, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:35.926213] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:35.926240] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302035926208) [2025-05-15 09:40:35.926254] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302035726096, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:35.926281] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12338197820, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:35.926296] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.926317] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:35.926335] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.926345] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.926346] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12338025327, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:35.926360] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.926362] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:35.926369] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:35.926377] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035926267) [2025-05-15 09:40:35.926380] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:35.926392] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:35.926403] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:35.926422] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.926431] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:35.926440] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302035926332) [2025-05-15 09:40:35.926577] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [4627][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=23] XA scheduler heartbeat task statistics(avg_time=2) [2025-05-15 09:40:35.927585] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:35.928328] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.928348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.928369] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.928384] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:35.928395] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.928407] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:35.928418] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.928428] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.928436] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302035928308, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:35.928449] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.928458] WDIAG [SHARE] get_valid_dest_pairs (ob_archive_persist_helper.cpp:427) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=8][errcode=-5627] failed to exec sql(ret=-5627, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:35.928529] WDIAG [ARCHIVE] load_archive_round_attr (ob_archive_persist_mgr.cpp:294) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=10][errcode=-5627] get valid dest pair failed(ret=-5627, tenant_id_=1002) [2025-05-15 09:40:35.928541] WDIAG [ARCHIVE] do_check_switch_archive_ (ob_archive_service.cpp:254) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=11][errcode=-5627] load archive round attr failed(ret=-5627, tenant_id=1002) [2025-05-15 09:40:35.929761] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.929777] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=15][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.929791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.929803] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.929814] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:35.929823] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.929833] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.929842] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302035929752, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:35.929852] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.929858] WDIAG [SHARE] get_valid_dest_pairs (ob_archive_persist_helper.cpp:427) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=6][errcode=-5627] failed to exec sql(ret=-5627, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:35.929896] WDIAG [ARCHIVE] load_archive_round_attr (ob_archive_persist_mgr.cpp:294) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=8][errcode=-5627] get valid dest pair failed(ret=-5627, tenant_id_=1002) [2025-05-15 09:40:35.929906] WDIAG [ARCHIVE] persist_archive_progress_ (ob_archive_persist_mgr.cpp:341) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] load archive round attr failed(ret=-5627, attr={key:{tenant_id:0, dest_no:-1}, incarnation:1, dest_id:0, round_id:0, state:{status:"INVALID"}, start_scn:{val:0, v:0}, checkpoint_scn:{val:0, v:0}, max_scn:{val:0, v:0}, compatible:{version:1}, base_piece_id:0, used_piece_id:0, piece_switch_interval:0, frozen_input_bytes:0, frozen_output_bytes:0, active_input_bytes:0, active_output_bytes:0, deleted_input_bytes:0, deleted_output_bytes:0, path:"", comment:""}) [2025-05-15 09:40:35.933388] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:35.933407] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.933422] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:35.934588] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=17] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts=0) [2025-05-15 09:40:35.934581] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4828430686, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:35.934607] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [4580][T1002_TimerWK3_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=18] update reserved snapshot finished(cost_ts=22, reserved_snapshot=0) [2025-05-15 09:40:35.934610] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:35.934622] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1aaefec2b0) [2025-05-15 09:40:35.934634] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4828430686, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:35.935016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.945126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.955235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.958756] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=34] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:35.958784] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:35.958796] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:35.958837] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=39] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:35.958854] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11798849180, flush_io_finished_ret_:0, flush_io_finished_round_:1510, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:35.959113] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=66] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:35.961794] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=83] tx gc loop thread is running(MTL_ID()=1) [2025-05-15 09:40:35.961814] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=20] try gc retain ctx [2025-05-15 09:40:35.961822] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=7] try to retry start_working [2025-05-15 09:40:35.961855] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=5] finish update min start scn(max_decided_scn={val:1747289697727835001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289694525675001, v:0}, keep_alive_scn_:{val:1747289694525675001, v:0}, update_ts_:1747302035961472}) [2025-05-15 09:40:35.962444] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=23] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:35.965347] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.970309] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4578][T1002_TimerWK1_][T1002][Y0-0000000000000000-0-0] [lt=12] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:35.970346] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4578][T1002_TimerWK1_][T1002][Y0-0000000000000000-0-0] [lt=30] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:35.970382] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4578][T1002_TimerWK1_][T1002][Y0-0000000000000000-0-0] [lt=34] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-05-15 09:40:35.970396] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4578][T1002_TimerWK1_][T1002][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=71) [2025-05-15 09:40:35.975461] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.975818] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:43, tid:3961}]) [2025-05-15 09:40:35.985583] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.994513] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4326][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=69] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1ac33de2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:35.995616] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDE-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.995637] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDE-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.995658] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDE-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.995673] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDE-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.995695] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:35.995741] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CDE-0-0] [lt=67][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.995751] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:35.995760] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.995772] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.995783] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302035995589, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:35.995797] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.995806] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:35.995884] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:35.997093] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDF-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.997110] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDF-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:35.997122] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDF-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:35.997134] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CDF-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:35.997145] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CDF-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:35.997157] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:35.997167] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:35.997175] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:35.997185] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302035997084, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:35.997197] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:35.997204] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:35.997243] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:35.998620] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:35.998907] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:35.998934] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:35.998952] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:35.998967] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:35.998982] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:35.998995] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:35.998991] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:35.999008] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:35.999011] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:35.999016] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:35.999020] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:35.999031] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:35.999042] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:35.999052] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:35.999068] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:35.999083] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:35.999126] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:36.003614] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.004563] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.004591] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036004557) [2025-05-15 09:40:36.004612] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302035804660, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.004653] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12338252512, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.004673] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.004696] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.004715] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.004729] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.004750] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12338252512, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.004764] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.004784] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.004801] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.004816] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.004834] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.004845] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.004857] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036004631) [2025-05-15 09:40:36.005804] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.008827] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.008837] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] ====== tenant freeze timer task ====== [2025-05-15 09:40:36.008850] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8262880, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.008867] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8262880, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.008881] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.008894] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=8262880, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.008910] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.008928] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.008945] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.008961] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.008973] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.008988] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.009006] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036009004, replica_locations:[]}) [2025-05-15 09:40:36.009024] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.009049] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:262, local_retry_times:262, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:36.009076] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.009090] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:36.009107] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:36.009117] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:36.009124] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:36.009140] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:36.009153] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.009197] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11799899674, cache_obj->added_lc()=false, cache_obj->get_object_id()=15785, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.010407] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:36.010434] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.010469] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.010484] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.010510] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.010524] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.010535] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.010549] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.010565] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.010580] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.010593] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.010605] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.010618] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.010632] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036010631, replica_locations:[]}) [2025-05-15 09:40:36.010649] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:36.010662] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.010675] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:36.010691] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:36.010701] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:36.010712] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:36.010730] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:36.010745] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:36.010753] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:36.010764] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:36.010774] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:36.010784] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:36.010798] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:36.010811] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:36.010822] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:36.010833] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:36.010843] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:36.010854] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:36.010865] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:36.010881] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:36.010893] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.010904] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:36.010915] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:36.010928] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:36.010937] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=263, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:36.010965] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=8260717, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=263, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.015914] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.016439] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.016460] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.016480] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.016506] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.016518] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.016529] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.016539] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.016549] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.016560] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302036016418, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.016574] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.016583] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.016645] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:36.016656] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:36.021625] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.021646] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.021662] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:36.022802] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:36.022821] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.022833] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:36.022850] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:36.022861] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:36.022872] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:36.023243] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4577][T1002_TimerWK0_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=19] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:36.023263] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4577][T1002_TimerWK0_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=14] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=82) [2025-05-15 09:40:36.026027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.026213] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:36.026248] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=12338298053, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.026266] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.026284] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.026299] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.026309] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.026324] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.026335] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.026343] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036026235) [2025-05-15 09:40:36.026396] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.026419] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036026391) [2025-05-15 09:40:36.026432] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302035826339, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.026455] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12338125560, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.026469] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.026486] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.026510] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.026521] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.026533] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.026542] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.026551] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036026445) [2025-05-15 09:40:36.033447] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:36.033467] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.033477] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:36.036140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.039008] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:36.046256] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.046458] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=42] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:36.046502] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:36.056367] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.062511] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.062534] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.062561] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:36.062574] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:36.062573] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC9694-0-0] [lt=34][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302036062545}) [2025-05-15 09:40:36.062587] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:36.062602] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:36.062612] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.062631] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:36.062641] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:36.062652] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:36.062661] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:36.062669] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:36.062676] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:36.062686] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:36.062698] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=9113, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:36.063038] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:36.063050] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.063066] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.063075] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:36.063084] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:36.063092] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:36.063100] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:36.063109] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.063122] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:36.065842] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=22] ====== check clog disk timer task ====== [2025-05-15 09:40:36.065863] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=16] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:36.066333] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4579][T1002_TimerWK2_Flush][T1002][Y0-0000000000000000-0-0] [lt=11] ====== traversal_flush timer task ====== [2025-05-15 09:40:36.066476] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.066406] INFO [MDS] flush (mds_table_mgr.cpp:161) [4579][T1002_TimerWK2_Flush][T1002][YB427F000001-000635291F9C6662-0-0] [lt=22] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1a6ea99150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:36.066541] INFO [MDS] flush (mds_table_mgr.cpp:161) [4579][T1002_TimerWK2_Flush][T1002][YB427F000001-000635291F9C6662-0-0] [lt=106] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1a6eb09150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:36.066603] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4579][T1002_TimerWK2_Flush][T1002][YB427F000001-000635291F9C6662-0-0] [lt=61] succeed to traversal_flush(ret=0, ls_cnt=2) [2025-05-15 09:40:36.069388] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:36.069417] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.069432] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:36.073936] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] new mysql sessid created(conn.sessid_=3221553005, support_ssl=false) [2025-05-15 09:40:36.074061] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221553005, sess.client_addr_="172.22.0.1:46150") [2025-05-15 09:40:36.074079] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553005, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.074221] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.074258] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.074293] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.074331] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=35][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.074349] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.074356] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.074383] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.074401] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553005, proxy_sessid=0) [2025-05-15 09:40:36.074429] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=10][errcode=0] server close connection(sessid=3221553005, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.074443] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553005, proxy_sessid=0) [2025-05-15 09:40:36.074450] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:36.074465] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906C-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553005, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.074538] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553005, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036073303, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.074565] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553005, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.074577] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553005, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036073303, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.074595] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553005) [2025-05-15 09:40:36.074614] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553005, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.076596] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.086706] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.096817] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.102275] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4318][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=35] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:36.103681] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.104525] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=26] ====== tenant freeze timer task ====== [2025-05-15 09:40:36.104639] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12338352679, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.104664] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.104732] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=67][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.104753] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.104764] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.104781] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12338352679, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.104794] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.104808] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.104820] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.104829] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.104840] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.104848] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.104856] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.106926] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.111088] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.111111] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8160610, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.111128] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8160610, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.111142] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.111154] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=8160610, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.111169] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.111186] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.111201] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.111216] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.111228] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.111242] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.111257] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036111255, replica_locations:[]}) [2025-05-15 09:40:36.111275] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.111294] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:263, local_retry_times:263, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:36.111316] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.111328] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:36.111346] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:36.111357] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:36.111363] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:36.111401] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:36.111416] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.111474] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11800001949, cache_obj->added_lc()=false, cache_obj->get_object_id()=15786, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.112657] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:36.112685] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.112722] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.112737] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.112750] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.112765] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.112773] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.112787] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.112802] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.112817] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.112829] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.112841] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.112855] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.112869] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036112868, replica_locations:[]}) [2025-05-15 09:40:36.112886] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:36.112899] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.112912] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:36.112927] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:36.112939] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:36.112950] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:36.112968] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:36.112982] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:36.112992] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:36.113003] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:36.113013] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:36.113024] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:36.113037] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:36.113049] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:36.113057] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:36.113067] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:36.113075] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:36.113083] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:36.113095] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:36.113108] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:36.113119] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.113131] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:36.113142] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:36.113154] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:36.113166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=264, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:36.113191] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=8158490, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=264, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.115251] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:36.115279] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:36.115296] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:36.115317] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=20] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:36.115856] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221553008, support_ssl=false) [2025-05-15 09:40:36.115857] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553007, support_ssl=false) [2025-05-15 09:40:36.115932] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221553007, sess.client_addr_="172.22.0.1:46160") [2025-05-15 09:40:36.115935] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553008, sess.client_addr_="172.22.0.1:46162") [2025-05-15 09:40:36.115950] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553008, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.115949] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553007, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.115978] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553009, support_ssl=false) [2025-05-15 09:40:36.116019] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553009, sess.client_addr_="172.22.0.1:46168") [2025-05-15 09:40:36.116033] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553009, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.116091] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.116133] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.116149] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.116166] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.116185] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.116193] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.116199] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.116207] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.116212] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.116225] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.116240] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.116242] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116251] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.116260] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553007, proxy_sessid=0) [2025-05-15 09:40:36.116279] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116295] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553008, proxy_sessid=0) [2025-05-15 09:40:36.116304] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.116306] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=18][errcode=0] server close connection(sessid=3221553007, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116318] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553007, proxy_sessid=0) [2025-05-15 09:40:36.116319] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=11][errcode=0] server close connection(sessid=3221553008, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116325] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.116329] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553008, proxy_sessid=0) [2025-05-15 09:40:36.116331] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:36.116338] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:36.116347] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=35][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.116343] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907D-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553007, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.116345] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AE-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553008, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.116369] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.116379] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.116386] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.116402] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116404] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553007, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036115461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.116419] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553009, proxy_sessid=0) [2025-05-15 09:40:36.116413] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553008, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036115461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.116427] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553007, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.116436] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553008, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.116441] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=9][errcode=0] server close connection(sessid=3221553009, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116438] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553007, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036115461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.116450] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553009, proxy_sessid=0) [2025-05-15 09:40:36.116445] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553008, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036115461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.116454] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553007) [2025-05-15 09:40:36.116458] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:36.116461] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553008) [2025-05-15 09:40:36.116468] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906D-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553009, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.116476] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553007, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.116478] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553008, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.116529] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=38] new mysql sessid created(conn.sessid_=3221553010, support_ssl=false) [2025-05-15 09:40:36.116531] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=39][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553009, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302036115461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.116558] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221553009, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.116564] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221553010, sess.client_addr_="172.22.0.1:46170") [2025-05-15 09:40:36.116580] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553010, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.116609] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553009, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302036115461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.116627] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553009) [2025-05-15 09:40:36.116639] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553009, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.116796] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.116813] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.116839] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.116855] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.116848] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553011, support_ssl=false) [2025-05-15 09:40:36.116871] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.116880] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.116886] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.116903] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116915] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553010, proxy_sessid=0) [2025-05-15 09:40:36.116923] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221553011, sess.client_addr_="172.22.0.1:46184") [2025-05-15 09:40:36.116935] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=8][errcode=0] server close connection(sessid=3221553010, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.116945] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553010, proxy_sessid=0) [2025-05-15 09:40:36.116954] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:36.116944] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f616db0, session_id:3221553011, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.116962] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90AF-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553010, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.117010] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553010, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036116519, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117031] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553010, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.117039] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553010, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036116519, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.117033] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553012, support_ssl=false) [2025-05-15 09:40:36.117052] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221553010) [2025-05-15 09:40:36.117046] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.117070] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553010, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.117088] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221553012, sess.client_addr_="172.22.0.1:46194") [2025-05-15 09:40:36.117104] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553012, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117179] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.117268] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=44] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.117284] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553013, support_ssl=false) [2025-05-15 09:40:36.117306] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=29][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.117318] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=38] new mysql sessid created(conn.sessid_=3221553014, support_ssl=false) [2025-05-15 09:40:36.117346] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.117352] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=44][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.117366] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.117374] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.117379] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.117378] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553014, sess.client_addr_="172.22.0.1:46216") [2025-05-15 09:40:36.117374] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553013, sess.client_addr_="172.22.0.1:46202") [2025-05-15 09:40:36.117400] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117394] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f616030, session_id:3221553014, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117401] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=35][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.117397] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553013, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117422] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.117431] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.117437] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.117452] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117452] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=47][errcode=-4018] get session fail(ret=-4018, sessid=3221553011, proxy_sessid=0) [2025-05-15 09:40:36.117465] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553012, proxy_sessid=0) [2025-05-15 09:40:36.117502] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=13][errcode=0] server close connection(sessid=3221553011, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117508] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=17][errcode=0] server close connection(sessid=3221553012, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117516] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553011, proxy_sessid=0) [2025-05-15 09:40:36.117522] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553012, proxy_sessid=0) [2025-05-15 09:40:36.117526] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:36.117537] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:36.117544] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906E-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553012, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.117541] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907E-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553011, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.117593] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.117609] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.117628] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.117651] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=24] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.117646] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.117663] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.117674] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.117672] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.117682] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.117641] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553012, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302036116519, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117668] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f616db0, session_id:3221553011, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036116519, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117688] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.117688] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.117696] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=48] kill and revert session(conn.sessid_=3221553012, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.117699] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.117703] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117697] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553011, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.117718] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f1a6f616db0, session_id:3221553011, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036116519, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.117719] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553013, proxy_sessid=0) [2025-05-15 09:40:36.117720] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553012, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302036116519, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.117736] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553011) [2025-05-15 09:40:36.117727] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117745] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=16][errcode=0] server close connection(sessid=3221553013, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117748] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] mark session id unused(conn.sessid_=3221553012) [2025-05-15 09:40:36.117749] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221553014, proxy_sessid=0) [2025-05-15 09:40:36.117755] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553013, proxy_sessid=0) [2025-05-15 09:40:36.117765] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:36.117758] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553011, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.117771] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC907F-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553013, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.117776] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=16][errcode=0] server close connection(sessid=3221553014, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.117790] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553014, proxy_sessid=0) [2025-05-15 09:40:36.117799] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221553015, support_ssl=false) [2025-05-15 09:40:36.117804] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:36.117793] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] connection close(sessid=3221553012, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.117814] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B0-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553014, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.117836] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553015, sess.client_addr_="172.22.0.1:46226") [2025-05-15 09:40:36.117817] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553013, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117848] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553015, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117850] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221553013, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.117871] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221553016, support_ssl=false) [2025-05-15 09:40:36.117867] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553013, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.117893] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] mark session id unused(conn.sessid_=3221553013) [2025-05-15 09:40:36.117924] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553016, sess.client_addr_="172.22.0.1:46238") [2025-05-15 09:40:36.117924] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] connection close(sessid=3221553013, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.117938] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553016, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117971] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f616030, session_id:3221553014, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.117988] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221553014, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.118027] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f616030, session_id:3221553014, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.118046] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553014) [2025-05-15 09:40:36.118062] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553014, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.118093] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.118116] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.118135] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.118155] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.118155] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.118164] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.118171] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.118176] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.118183] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.118187] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.118194] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221553017, support_ssl=false) [2025-05-15 09:40:36.118200] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.118200] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553015, proxy_sessid=0) [2025-05-15 09:40:36.118208] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.118216] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.118219] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=8][errcode=0] server close connection(sessid=3221553015, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.118227] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] sm conn init succ(conn.sessid_=3221553017, sess.client_addr_="172.22.0.1:46240") [2025-05-15 09:40:36.118231] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553015, proxy_sessid=0) [2025-05-15 09:40:36.118231] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.118238] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a6f620030, session_id:3221553017, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.118246] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:36.118247] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553016, proxy_sessid=0) [2025-05-15 09:40:36.118254] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC906F-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553015, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.118265] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=7][errcode=0] server close connection(sessid=3221553016, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.118276] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553016, proxy_sessid=0) [2025-05-15 09:40:36.118283] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:36.118290] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9080-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553016, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.118311] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553015, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.118347] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553016, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.118364] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] kill and revert session(conn.sessid_=3221553015, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.118374] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221553016, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.118382] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553015, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.118396] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553015) [2025-05-15 09:40:36.118409] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553015, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.118428] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553016, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.118441] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553016) [2025-05-15 09:40:36.118450] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553016, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.118476] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.118507] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.118523] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.118542] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.118551] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.118562] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.118575] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.118591] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553017, proxy_sessid=0) [2025-05-15 09:40:36.118611] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=7][errcode=0] server close connection(sessid=3221553017, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.118623] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553017, proxy_sessid=0) [2025-05-15 09:40:36.118630] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:36.118636] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B1-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553017, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.118681] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f620030, session_id:3221553017, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.118703] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553017, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.118712] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f620030, session_id:3221553017, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036117578, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.118726] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553017) [2025-05-15 09:40:36.118749] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553017, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.120174] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:36.121692] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.121720] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.121758] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=35][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:36.126277] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.126306] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036126269) [2025-05-15 09:40:36.126326] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302035926267, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.126366] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12338398179, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.126388] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.126414] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.126455] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.126477] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036126451) [2025-05-15 09:40:36.126501] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302036026444, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.126523] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12338225686, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.126540] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.126559] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.127162] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.127872] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:36.128252] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:36.128279] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-05-15 09:40:36.128295] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:36.128313] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:36.128333] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11799018671, flush_io_finished_ret_:0, flush_io_finished_round_:1510, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:36.137286] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.147400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.157514] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.167626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.173686] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.174336] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.174751] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.174934] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.177737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.187847] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.189231] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=15] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:36.189262] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:36.189280] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:36.189298] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:36.189314] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:36.189327] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:36.189340] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:36.189354] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:36.189362] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:36.189374] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:36.189383] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:36.189391] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=7][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:36.189404] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:36.189416] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:36.189435] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:36.189448] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.189463] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.189478] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:36.189503] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=22][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:36.189519] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:36.189532] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:36.189556] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=18][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:36.189579] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.189592] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.189598] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:36.189619] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:36.189635] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C646F-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.189650] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:36.189664] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.189674] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:36.189687] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:36.189700] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302036188971, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.189715] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:36.189723] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.189829] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=21][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:36.189846] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=16][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:36.189860] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=13][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.189868] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=8][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.189882] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=11][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:36.189897] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=13][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:36.189905] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C646F-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.191587] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:36.197956] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.203741] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.204679] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.204721] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036204672) [2025-05-15 09:40:36.204739] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302036004630, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.204796] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=40] get wrs ts(ls_id={id:1}, delta=12338452793, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.204815] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.204850] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12338452793, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.204865] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.208078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.213341] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.213371] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8058364, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.213392] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=8058364, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.213408] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.213420] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=8058364, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.213438] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.213457] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.213475] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.213504] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.213514] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.213531] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.213551] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036213549, replica_locations:[]}) [2025-05-15 09:40:36.213573] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.213595] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.213620] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.213633] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:36.213672] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.213731] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11800104208, cache_obj->added_lc()=false, cache_obj->get_object_id()=15787, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.215090] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.215113] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.215130] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.215146] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.215154] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.215170] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.215187] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.215204] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.215220] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.215229] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.215246] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.215262] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036215261, replica_locations:[]}) [2025-05-15 09:40:36.215346] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=8056336, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=265, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.216530] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.218194] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.221745] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.222921] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.226340] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12338498283, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.226362] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.226523] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12338325790, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.226544] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.228315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.233566] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.233593] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302036233516, register_time_us=-1, curr_time_us=1747302036233591) [2025-05-15 09:40:36.238436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.239097] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:36.248927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.254870] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.254890] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:36.254900] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:36.254918] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:36.254933] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=10398720) [2025-05-15 09:40:36.254962] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=28, size_used=0, mem_used=0) [2025-05-15 09:40:36.254989] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:36.258934] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:36.258967] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=26] temporary file module use no blocks [2025-05-15 09:40:36.258976] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:36.258996] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:36.259016] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11799149353, flush_io_finished_ret_:0, flush_io_finished_round_:1510, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:36.259588] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.267652] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:36.269078] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=14][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:36.269101] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=22][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.269118] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.269137] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.269151] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.269164] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.269180] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.269198] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.269215] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.269230] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.269242] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.269259] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.269288] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:36.269303] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=11] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:36.269706] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.271005] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:36.271032] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:36.271046] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:36.271059] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:36.271072] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:36.271083] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:36.271095] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:36.271105] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:36.271115] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:36.271124] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:36.271133] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:36.271143] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:36.271152] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:36.271161] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:36.271180] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:36.271192] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.271205] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.271216] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:36.271226] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:36.271239] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:36.271249] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:36.271268] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:36.271289] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.271298] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.271305] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:36.271323] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:36.271334] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.271344] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:36.271355] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.271365] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:36.271376] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:36.271386] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca3b0, start=1747302036270764, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.271398] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:36.271407] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.271503] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:36.271517] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=12][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:36.271533] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=16][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.271541] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=7][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.271552] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:36.271563] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=9][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:36.271572] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BD-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.276429] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:35, tid:4630}]) [2025-05-15 09:40:36.279435] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C6441-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.279822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.280717] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291D5C6442-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.289940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.297567] WDIAG load_file_to_string (utility.h:673) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:36.297589] WDIAG get_ethernet_speed (utility.cpp:575) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:36.297600] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:36.297614] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:36.300059] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.303811] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.304753] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12338552945, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.304781] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.304808] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.304829] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.304843] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.304859] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12338552945, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.304874] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.304891] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.304905] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.304912] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.304927] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.304940] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.304949] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036304731) [2025-05-15 09:40:36.310176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.315478] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.315513] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7956220, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.315530] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7956220, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.315545] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.315555] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=7956220, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.315570] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.315588] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.315606] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.315622] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.315636] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.315654] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.315672] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036315670, replica_locations:[]}) [2025-05-15 09:40:36.315692] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.315719] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.315739] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:36.315785] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.315832] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11800206311, cache_obj->added_lc()=false, cache_obj->get_object_id()=15788, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.316560] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.316988] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.317011] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.317029] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.317040] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.317053] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.317069] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.317086] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.317101] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.317116] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.317125] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.317141] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.317158] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036317156, replica_locations:[]}) [2025-05-15 09:40:36.317225] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7954457, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=266, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.320294] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.322011] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4291][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=8] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:36.322017] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4289][T1_TimerWK0_Flush][T1][Y0-0000000000000000-0-0] [lt=20] ====== traversal_flush timer task ====== [2025-05-15 09:40:36.322039] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4291][T1_TimerWK2_TabletShell][T1][Y0-0000000000000000-0-0] [lt=20] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=71) [2025-05-15 09:40:36.322081] INFO [MDS] flush (mds_table_mgr.cpp:161) [4289][T1_TimerWK0_Flush][T1][YB427F000001-000635291C0C63BF-0-0] [lt=18] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1aaf36f150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:36.322191] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4289][T1_TimerWK0_Flush][T1][YB427F000001-000635291C0C63BF-0-0] [lt=104] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-15 09:40:36.326401] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:36.326425] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.326449] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036326394) [2025-05-15 09:40:36.326465] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302036126342, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.326503] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12338598543, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.326521] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.326542] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.326556] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.326569] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.326582] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.326576] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.326593] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.326602] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036326479) [2025-05-15 09:40:36.326608] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036326572) [2025-05-15 09:40:36.326624] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302036126511, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.326650] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12338426050, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.326664] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.326680] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.326694] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.326703] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.326715] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.326724] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.326730] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036326640) [2025-05-15 09:40:36.328151] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:36.330408] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.332282] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4292][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=0] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:36.332309] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4292][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=21] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:36.332325] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4292][T1_TimerWK3_][T1][Y0-0000000000000000-0-0] [lt=14] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=71) [2025-05-15 09:40:36.333435] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5256, clean_start_pos=0, clean_num=200000) [2025-05-15 09:40:36.334780] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] [thread pool stat](pool_stat={pool:0x5653d3b08200, last_check_time:1747302036334778, last_idle_time:6070104296, last_thread_count:4}, *pool={name:TimerWK, this:0x5653d3b08200, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:6070104296, tenant_id:500}, qsize=-4, current_idle_time=6070104296) [2025-05-15 09:40:36.334808] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=27] [thread pool stat](pool_stat={pool:0x7f1ae77ad170, last_check_time:1747302036334807, last_idle_time:1515998294, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f1ae77ad170, min_thread_cnt:1, max_thread_cnt:8, running_thread_cnt:0, threads_idle_time:1515998294, tenant_id:500}, qsize=-1, current_idle_time=1515998294) [2025-05-15 09:40:36.334822] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] [thread pool stat](pool_stat={pool:0x7f1ae77adf70, last_check_time:1747302036334822, last_idle_time:1520194160, last_thread_count:1}, *pool={name:IO_SYNC_CH, this:0x7f1ae77adf70, min_thread_cnt:1, max_thread_cnt:96, running_thread_cnt:0, threads_idle_time:1520194160, tenant_id:500}, qsize=-1, current_idle_time=1520194160) [2025-05-15 09:40:36.334837] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] [thread pool stat](pool_stat={pool:0x7f1ae73fe3b0, last_check_time:1747302036334836, last_idle_time:4829717803, last_thread_count:5}, *pool={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4829717803, tenant_id:1}, qsize=-3, current_idle_time=4829717803) [2025-05-15 09:40:36.334851] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] [thread pool stat](pool_stat={pool:0x7f1aaa0b0e70, last_check_time:1747302036334850, last_idle_time:6045936463, last_thread_count:4}, *pool={name:TimerWK, this:0x7f1aaa0b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:6045936463, tenant_id:1001}, qsize=-4, current_idle_time=6045936463) [2025-05-15 09:40:36.334861] INFO [SHARE] run1 (ob_dynamic_thread_pool.cpp:609) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=11] [thread pool stat](pool_stat={pool:0x7f1a8f8b0e70, last_check_time:1747302036334861, last_idle_time:6071586602, last_thread_count:4}, *pool={name:TimerWK, this:0x7f1a8f8b0e70, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:0, threads_idle_time:6071586602, tenant_id:1002}, qsize=-4, current_idle_time=6071586602) [2025-05-15 09:40:36.340526] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.350644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.354440] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:36.354478] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.354505] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:36.354519] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:36.360750] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.362601] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:36.362674] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:36.362740] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:36.362799] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:36.362828] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:18064, total:18074) [2025-05-15 09:40:36.362863] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:36.362940] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=73] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:18064, total:18074) [2025-05-15 09:40:36.362968] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:36.362986] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:36.362999] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:36.363019] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] successfully alloc block(macro_id={[ver=1,mode=0,seq=628771][2nd=1898]}) [2025-05-15 09:40:36.370878] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.376651] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:82, tid:4602}]) [2025-05-15 09:40:36.380997] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.391102] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.392913] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4292][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=12] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}) [2025-05-15 09:40:36.395860] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.395873] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.395931] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=23] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.396665] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.396696] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=35] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.396795] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=45] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.396940] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:36.401209] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.403873] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=32][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.404796] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.404828] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036404789) [2025-05-15 09:40:36.404848] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302036204778, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.404882] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12338653533, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.404901] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.404921] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.404937] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.404947] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.404963] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12338653533, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.404976] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.404989] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.405000] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.405009] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.405021] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.405030] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.405039] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036404865) [2025-05-15 09:40:36.411315] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.413108] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.416731] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.416754] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.416777] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.416793] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.416806] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.416820] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.416832] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.416842] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.416853] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302036416703, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.416867] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.416878] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.416966] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:36.416991] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=25][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:36.417005] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:36.417018] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:36.417349] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.417370] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7854347, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.417387] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7854347, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.417401] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.417415] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=7854347, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.417433] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.417446] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.417457] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.417468] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.417479] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.417505] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.417521] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036417520, replica_locations:[]}) [2025-05-15 09:40:36.417539] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.417563] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.417572] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:36.417598] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.417651] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11800308128, cache_obj->added_lc()=false, cache_obj->get_object_id()=15789, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.418787] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.418807] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.418822] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.418834] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.418844] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.418857] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.418871] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.418884] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.418897] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.418907] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.418918] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.418930] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036418930, replica_locations:[]}) [2025-05-15 09:40:36.418989] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7852692, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=267, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.421424] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.426470] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12338698037, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.426504] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=31] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.426525] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.426541] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.426553] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.426569] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.426581] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.426589] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036426455) [2025-05-15 09:40:36.426644] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12338525544, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.426664] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.426682] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.426696] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.426707] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.426721] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.426731] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.426741] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036426632) [2025-05-15 09:40:36.432563] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.433345] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=0] ====== checkpoint timer task ====== [2025-05-15 09:40:36.433990] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=16] succ to get rec scn(*this={this:0x7f1ab0204030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300517859523, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1303}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.434031] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=41] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a96ba2080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a96ba2080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.434107] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=48] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1aaf36f150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:36.434194] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=84] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1aaf36e1b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1aaf379010, memtable_head:0, memtable_tail:1, t3m:0x7f1ac66cc030, tables:[0x7f1a96b92080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1aaf37ad10, ls_tablet_svr:0x7f1aaf36e190}, min_rec_scn={val:1747274404592930000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:36.434230] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-15 09:40:36.434246] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=0, rec_log_ts={val:1747289692924660001, v:0}) [2025-05-15 09:40:36.434259] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.434267] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=6] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.434282] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:1747277617103769000, v:0}, flushed_scn_={val:1746499972147257000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:1747277617937058000, v:0}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:36.434304] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=18] rec_scn of ObLockMemtable is (rec_scn_={val:1747277617103769000, v:0}, flushed_scn_={val:1746499972147257000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:1747277617937058000, v:0}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:36.434321] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274404592930000, v:0}, checkpoint_scn_in_ls_meta={val:1747274404592930000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:36.434343] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2818400256}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2818400256}, lsn={lsn:2818400256}) [2025-05-15 09:40:36.434368] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2818400256}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:36.434381] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4289][T1_TimerWK0_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=11] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-15 09:40:36.439192] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:36.441316] WDIAG [SHARE] runTimerTask (ob_all_server_tracer.cpp:1001) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4036] sys tenant schema is not ready(ret=-4036, ret="OB_NEED_RETRY") [2025-05-15 09:40:36.442670] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.452780] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.456097] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:133) [3954][TimerWK1_ServerTracerTim][T0][YB427F000001-000635291F2C618E-0-0] [lt=19][errcode=-4002] invalid argument, empty server list(ret=-4002) [2025-05-15 09:40:36.456118] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:378) [3954][TimerWK1_ServerTracerTim][T0][YB427F000001-000635291F2C618E-0-0] [lt=20][errcode=-4002] refresh sever list failed(ret=-4002) [2025-05-15 09:40:36.456126] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:252) [3954][TimerWK1_ServerTracerTim][T0][YB427F000001-000635291F2C618E-0-0] [lt=8][errcode=-4002] refresh alive server list failed(ret=-4002) [2025-05-15 09:40:36.462887] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.472996] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.483108] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.493212] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.499109] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.499155] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5323) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=0] is_in_sync(ret=-4721, palf_id=1, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1747289697897144000, v:0}) [2025-05-15 09:40:36.499176] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.499184] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5323) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=1] is_in_sync(ret=-4721, palf_id=1001, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1747289697898202000, v:0}) [2025-05-15 09:40:36.499199] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.499208] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.499217] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.499231] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:36.499249] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:36.499267] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:36.499281] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:36.499320] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:36.503322] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.503923] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.504868] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12338753002, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.504896] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.504938] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.504958] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.504970] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.504986] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12338753002, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.504999] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.505014] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.505027] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.505034] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.505048] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.505056] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.505066] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036504849) [2025-05-15 09:40:36.507236] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C6664-0-0] [lt=16][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.507257] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C6664-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.507279] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C6664-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.507298] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C6664-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.507312] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6664-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.507326] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:36.507340] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.507349] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.507362] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302036507201, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:36.507377] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.507386] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:36.507465] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1002, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:36.507524] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=58][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:36.509050] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=8][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.509081] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=31][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.509091] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.509104] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=12][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.509116] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=12][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:36.509147] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=14][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:36.509162] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.509174] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C6665-0-0] [lt=11][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5157, ret=-5157) [2025-05-15 09:40:36.509186] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:36.509200] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.509209] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:36.509221] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78ca3b0, start=1747302036509022, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:36.509234] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:36.509245] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] execute sql failed(ret=-5157, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:36.509316] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to select for update snapshot_gc_scn(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.509332] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5157] failed to load updated info(ret=-5157) [2025-05-15 09:40:36.509345] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to try update info(tmp_ret=-5157, tmp_ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:36.510640] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=14] ====== checkpoint timer task ====== [2025-05-15 09:40:36.511298] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=17] succ to get rec scn(*this={this:0x7f1a69e04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300524716821, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:726}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.511340] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=42] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a690fa080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a690fa080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.511396] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=30] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1a6ea99150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:36.511476] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=78] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1a6ea981b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1a6eaa3010, memtable_head:0, memtable_tail:1, t3m:0x7f1a8f8ea030, tables:[0x7f1a690ea080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1a6eaa4d10, ls_tablet_svr:0x7f1a6ea98190}, min_rec_scn={val:1747274406063927000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:36.511524] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=41] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1747274406063927000, v:0}) [2025-05-15 09:40:36.511542] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=16] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.511556] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.511571] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=6] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:36.511594] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=18] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:36.511612] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274406063927000, v:0}, checkpoint_scn_in_ls_meta={val:1747274406063927000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:36.511636] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:872361984}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:872361984}, lsn={lsn:872361984}) [2025-05-15 09:40:36.511662] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=26] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:872361984}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:36.512257] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x7f1a69e76030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300524741888, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:5}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.512285] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=27] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a690fb390, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a690fb390, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:36.512324] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1a6eb09150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:36.512386] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=61] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-05-15 09:40:36.512396] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=9] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1747274672721723000, v:0}) [2025-05-15 09:40:36.512406] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-05-15 09:40:36.512425] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=18] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-05-15 09:40:36.512441] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=15] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747289697751194001, v:0}, checkpoint_scn_in_ls_meta={val:1747289697751194001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-05-15 09:40:36.512455] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:738152448}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1746497901775524000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:738152448}, lsn={lsn:738152448}) [2025-05-15 09:40:36.512475] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:738152448}, ls->get_ls_id()={id:1001}) [2025-05-15 09:40:36.512482] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4578][T1002_TimerWK1_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=6] succeed to update_clog_checkpoint(ret=0, ls_cnt=2) [2025-05-15 09:40:36.513436] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.516672] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.516694] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.516710] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.516742] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=31][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.516755] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.516767] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.516780] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.516788] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.516799] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302036516658, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.516813] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.516821] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.516870] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:36.516882] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:36.516894] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:36.516906] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:36.519129] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.519151] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7752572, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.519171] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7752572, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.519182] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.519195] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=7752572, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.519208] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.519224] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.519239] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.519254] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.519264] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.519278] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.519294] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036519292, replica_locations:[]}) [2025-05-15 09:40:36.519312] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.519338] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.519348] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:36.519388] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.519441] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11800409915, cache_obj->added_lc()=false, cache_obj->get_object_id()=15790, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.520761] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.520782] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.520796] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.520809] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.520818] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.520831] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.520845] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.520857] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.520870] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.520881] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.520895] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.520907] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036520906, replica_locations:[]}) [2025-05-15 09:40:36.520972] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7750710, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=268, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.523560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.526520] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.526559] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036526513) [2025-05-15 09:40:36.526576] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302036326479, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.526607] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12338798597, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.526626] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.526648] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.526665] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.526676] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.526692] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.526692] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.526703] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.526712] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036526595) [2025-05-15 09:40:36.526711] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036526689) [2025-05-15 09:40:36.526723] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302036326635, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.526743] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12338626104, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.526758] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.526775] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.526786] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.526796] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.526812] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.526822] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.526828] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036526732) [2025-05-15 09:40:36.530336] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.530353] WDIAG [SERVER] read_tenant_wa_percentage (ob_tenant_duty_task.cpp:169) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.530361] WDIAG [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:217) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] read variable tenant work area percentage fail(ret=-5627) [2025-05-15 09:40:36.530371] WDIAG [SERVER] update_all_tenants (ob_tenant_duty_task.cpp:53) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] update tenant work area memory fail(ret=-5627) [2025-05-15 09:40:36.530379] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.530390] WDIAG [SERVER] read_obj (ob_tenant_duty_task.cpp:269) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] get schema guard failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.530396] WDIAG [SERVER] read_int64 (ob_tenant_duty_task.cpp:288) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] get object from tenant system variable fail(ret=-5627) [2025-05-15 09:40:36.530407] WDIAG [SERVER] update_tenant_sql_throttle (ob_tenant_duty_task.cpp:106) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] get tenant sql throttle metrics fail(ret=-5627) [2025-05-15 09:40:36.530413] WDIAG [SERVER] update_all_tenants (ob_tenant_duty_task.cpp:59) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] update tenant sql throttle fail(ret=-5627) [2025-05-15 09:40:36.530431] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.530443] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.530449] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.530456] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.530464] WDIAG [SERVER] read_tenant_wa_percentage (ob_tenant_duty_task.cpp:169) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:36.530470] WDIAG [SERVER] update_tenant_wa_percentage (ob_tenant_duty_task.cpp:217) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=5][errcode=-5157] read variable tenant work area percentage fail(ret=-5157) [2025-05-15 09:40:36.530478] WDIAG [SERVER] update_all_tenants (ob_tenant_duty_task.cpp:53) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] update tenant work area memory fail(ret=-5157) [2025-05-15 09:40:36.530484] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.530509] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.530515] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.530524] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.530531] WDIAG [SERVER] read_obj (ob_tenant_duty_task.cpp:269) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] get schema guard failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.530540] WDIAG [SERVER] read_int64 (ob_tenant_duty_task.cpp:288) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] get object from tenant system variable fail(ret=-5157) [2025-05-15 09:40:36.530546] WDIAG [SERVER] update_tenant_sql_throttle (ob_tenant_duty_task.cpp:106) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] get tenant sql throttle metrics fail(ret=-5157) [2025-05-15 09:40:36.530554] WDIAG [SERVER] update_all_tenants (ob_tenant_duty_task.cpp:59) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] update tenant sql throttle fail(ret=-5157) [2025-05-15 09:40:36.533673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.533681] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:36.541647] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=39] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:36.542083] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:36.543783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.552684] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4462][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=6] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:36.553915] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.555417] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:36.555471] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=28] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:36.563534] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.563575] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:36.563593] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:36.563614] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:36.563628] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:36.563646] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:36.563997] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.564009] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:36.564021] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:36.564026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.572380] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6603-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.572402] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6603-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.572424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6603-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.572446] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6603-0-0] [lt=20][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.572459] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.572471] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.572484] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.572499] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=14][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.572512] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302036572355, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.572525] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.572533] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:36.572607] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=12][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:36.572621] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=12][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:36.572634] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=13][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:36.572642] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=7][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:36.572656] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=11][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:36.572669] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=11][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:36.572680] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6603-0-0] [lt=11][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:36.574135] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.575181] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553029, support_ssl=false) [2025-05-15 09:40:36.575305] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221553029, sess.client_addr_="172.22.0.1:46244") [2025-05-15 09:40:36.575323] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553029, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.575558] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.575595] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.575627] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.575662] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=33][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.575679] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.575687] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.575708] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.575726] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553029, proxy_sessid=0) [2025-05-15 09:40:36.575759] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=15][errcode=0] server close connection(sessid=3221553029, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.575772] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553029, proxy_sessid=0) [2025-05-15 09:40:36.575780] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:36.575794] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9070-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553029, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.575846] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553029, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036575144, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.575874] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221553029, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.575895] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553029, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036575144, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.575928] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=32] mark session id unused(conn.sessid_=3221553029) [2025-05-15 09:40:36.575947] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553029, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.584247] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.596521] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.603980] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.604916] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.604949] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036604908) [2025-05-15 09:40:36.604980] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302036404859, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.605019] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12338853595, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.605039] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.605063] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.605081] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.605093] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.605110] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12338853595, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.605126] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.605139] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.605148] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.605158] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.605169] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.605180] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.605191] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036604994) [2025-05-15 09:40:36.606636] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.616747] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.616771] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.616791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.616811] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.616829] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.616842] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.616855] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.616864] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.616874] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.616887] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302036616753, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.616900] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.616908] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.616969] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:36.616981] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:36.616992] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:36.617003] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:36.617302] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221553031, support_ssl=false) [2025-05-15 09:40:36.617368] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553031, sess.client_addr_="172.22.0.1:46256") [2025-05-15 09:40:36.617383] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553031, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.617449] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553032, support_ssl=false) [2025-05-15 09:40:36.617478] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553032, sess.client_addr_="172.22.0.1:46272") [2025-05-15 09:40:36.617503] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553032, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.617618] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.617646] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.617681] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.617711] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.617719] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.617722] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.617731] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.617740] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=8] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.617766] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.617783] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.617799] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.617792] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=46] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.617807] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.617814] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221553031, proxy_sessid=0) [2025-05-15 09:40:36.617833] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.617848] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=12][errcode=0] server close connection(sessid=3221553031, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.617854] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553032, proxy_sessid=0) [2025-05-15 09:40:36.617859] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553031, proxy_sessid=0) [2025-05-15 09:40:36.617867] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:36.617877] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9081-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553031, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.617885] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] new mysql sessid created(conn.sessid_=3221553033, support_ssl=false) [2025-05-15 09:40:36.617892] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=10][errcode=0] server close connection(sessid=3221553032, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.617912] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=20][errcode=-4018] get session fail(ret=-4018, sessid=3221553032, proxy_sessid=0) [2025-05-15 09:40:36.617922] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:36.617930] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553031, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036617498, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.617937] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B2-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553032, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.617953] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553031, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.617964] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553031, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036617498, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.617969] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221553033, sess.client_addr_="172.22.0.1:46288") [2025-05-15 09:40:36.617982] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553031) [2025-05-15 09:40:36.617990] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553033, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.617997] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553031, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.618037] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553032, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036617498, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.618056] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553032, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.618077] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553032, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302036617498, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.618097] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553032) [2025-05-15 09:40:36.618106] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553032, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.618209] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.618257] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.618272] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.618289] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.618304] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.618312] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.618333] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.618350] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553033, proxy_sessid=0) [2025-05-15 09:40:36.618370] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=9][errcode=0] server close connection(sessid=3221553033, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.618385] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553033, proxy_sessid=0) [2025-05-15 09:40:36.618392] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:36.618393] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553034, support_ssl=false) [2025-05-15 09:40:36.618408] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9071-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553033, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.618460] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553033, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036617498, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.618468] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221553034, sess.client_addr_="172.22.0.1:46302") [2025-05-15 09:40:36.618499] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553033, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.618515] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553033, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036617498, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.618501] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f620db0, session_id:3221553034, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.618531] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553033) [2025-05-15 09:40:36.618553] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221553033, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.618729] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.618773] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.618786] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.618801] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.618811] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.618820] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.618835] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.618847] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553034, proxy_sessid=0) [2025-05-15 09:40:36.618872] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=13][errcode=0] server close connection(sessid=3221553034, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.618882] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553034, proxy_sessid=0) [2025-05-15 09:40:36.618889] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:36.618896] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9082-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553034, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.618926] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553035, support_ssl=false) [2025-05-15 09:40:36.618975] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553035, sess.client_addr_="172.22.0.1:46304") [2025-05-15 09:40:36.618993] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553035, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.618971] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f620db0, session_id:3221553034, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619019] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=42] kill and revert session(conn.sessid_=3221553034, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.619023] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553036, support_ssl=false) [2025-05-15 09:40:36.619033] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a6f620db0, session_id:3221553034, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.619055] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553034) [2025-05-15 09:40:36.619059] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553036, sess.client_addr_="172.22.0.1:46310") [2025-05-15 09:40:36.619076] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553036, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619089] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221553034, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.619138] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221553037, support_ssl=false) [2025-05-15 09:40:36.619190] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221553038, support_ssl=false) [2025-05-15 09:40:36.619244] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553037, sess.client_addr_="172.22.0.1:46316") [2025-05-15 09:40:36.619245] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221553038, sess.client_addr_="172.22.0.1:46326") [2025-05-15 09:40:36.619261] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f620030, session_id:3221553037, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619262] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553038, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619311] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.619348] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.619376] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.619382] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.619404] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.619412] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=29][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.619427] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.619421] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.619441] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.619442] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.619450] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.619458] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.619460] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619476] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553036, proxy_sessid=0) [2025-05-15 09:40:36.619479] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619503] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=23][errcode=-4018] get session fail(ret=-4018, sessid=3221553035, proxy_sessid=0) [2025-05-15 09:40:36.619514] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.619518] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=24][errcode=0] server close connection(sessid=3221553036, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619522] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=8][errcode=0] server close connection(sessid=3221553035, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619529] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553036, proxy_sessid=0) [2025-05-15 09:40:36.619534] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553035, proxy_sessid=0) [2025-05-15 09:40:36.619540] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:36.619543] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.619546] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:36.619558] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.619552] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B3-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553036, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.619559] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9072-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553035, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.619572] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.619587] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.619596] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.619591] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.619613] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619625] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553037, proxy_sessid=0) [2025-05-15 09:40:36.619626] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553036, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619645] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=10][errcode=0] server close connection(sessid=3221553037, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619646] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.619657] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553037, proxy_sessid=0) [2025-05-15 09:40:36.619665] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:36.619659] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.619669] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221553039, support_ssl=false) [2025-05-15 09:40:36.619675] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.619660] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553035, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619683] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.619679] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9083-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553037, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.619691] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221553036, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.619693] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.619701] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553039, sess.client_addr_="172.22.0.1:46332") [2025-05-15 09:40:36.619698] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553040, support_ssl=false) [2025-05-15 09:40:36.619710] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553035, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.619712] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619714] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553039, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619721] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553036, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.619726] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553038, proxy_sessid=0) [2025-05-15 09:40:36.619720] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f620030, session_id:3221553037, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619740] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553036) [2025-05-15 09:40:36.619748] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221553037, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.619748] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=12][errcode=0] server close connection(sessid=3221553038, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.619749] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221553040, sess.client_addr_="172.22.0.1:46346") [2025-05-15 09:40:36.619758] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f620030, session_id:3221553037, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.619762] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553038, proxy_sessid=0) [2025-05-15 09:40:36.619764] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553040, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619774] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553037) [2025-05-15 09:40:36.619774] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:36.619775] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553036, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.619786] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B4-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553038, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.619793] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553041, support_ssl=false) [2025-05-15 09:40:36.619789] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553037, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.619799] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553035, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.619819] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553035) [2025-05-15 09:40:36.619836] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553041, sess.client_addr_="172.22.0.1:46360") [2025-05-15 09:40:36.619834] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553035, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.619833] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553038, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619859] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553038, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.619860] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553041, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.619872] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553038, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302036618557, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.619894] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553038) [2025-05-15 09:40:36.619936] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] connection close(sessid=3221553038, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.619964] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.620001] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.619992] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=41] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.620018] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.620032] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.619964] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [4090][Blacklist][T0][Y0-0000000000000000-0-0] [lt=32] blacklist_loop exec finished(cost_time=31, is_enabled=true, send_cnt=0) [2025-05-15 09:40:36.620032] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.620052] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.620049] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.620061] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.620065] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.620074] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.620073] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.620077] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:36.620081] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.620087] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553040, proxy_sessid=0) [2025-05-15 09:40:36.620095] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.620107] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553039, proxy_sessid=0) [2025-05-15 09:40:36.620109] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:36.620117] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=20][errcode=0] server close connection(sessid=3221553040, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.620124] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=7][errcode=0] server close connection(sessid=3221553039, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.620132] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553040, proxy_sessid=0) [2025-05-15 09:40:36.620122] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:36.620135] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553039, proxy_sessid=0) [2025-05-15 09:40:36.620142] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:36.620141] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:36.620143] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:36.620149] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9084-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553039, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.620152] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9073-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553040, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.620159] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:36.620174] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:36.620194] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.620209] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553041, proxy_sessid=0) [2025-05-15 09:40:36.620202] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553040, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302036619618, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.620200] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553039, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302036619618, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.620223] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553040, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.620227] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=8][errcode=0] server close connection(sessid=3221553041, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.620237] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553041, proxy_sessid=0) [2025-05-15 09:40:36.620234] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553040, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302036619618, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.620237] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] kill and revert session(conn.sessid_=3221553039, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.620245] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:36.620251] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553040) [2025-05-15 09:40:36.620253] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553039, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302036619618, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.620255] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B5-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553041, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:36.620275] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221553039) [2025-05-15 09:40:36.620264] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553040, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.620288] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553039, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.620298] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553041, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302036619618, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:36.620316] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553041, proxy_sessid=0, ret=0) [2025-05-15 09:40:36.620340] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553041, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302036619618, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:36.620363] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221553041) [2025-05-15 09:40:36.620374] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553041, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:36.621136] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.621163] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7650576, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.621184] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7650576, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.621200] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.621219] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=7650576, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.621236] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:36.621259] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:36.621276] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.621296] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.621309] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.621323] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.621345] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036621343, replica_locations:[]}) [2025-05-15 09:40:36.621367] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.621397] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.621410] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:36.621448] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.621525] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11800511998, cache_obj->added_lc()=false, cache_obj->get_object_id()=15791, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:36.622933] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.622955] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.622972] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.622987] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:36.622997] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:36.623018] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.623034] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.623043] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.623060] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.623075] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036623075, replica_locations:[]}) [2025-05-15 09:40:36.623149] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7648533, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=269, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.626603] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12338898134, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.626633] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.626661] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.626683] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.626697] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.626717] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.626730] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.626737] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036626575) [2025-05-15 09:40:36.626753] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.626782] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036626745) [2025-05-15 09:40:36.626801] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302036526732, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.626827] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12338725641, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.626844] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.626864] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.626875] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.626866] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.626888] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.626899] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.626911] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.626918] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036626816) [2025-05-15 09:40:36.637001] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.639284] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=25] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:36.647119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.655991] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EAE95-0-0] [lt=51][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1001, ls_id:{id:1}, add_timestamp:1747300526118711}) [2025-05-15 09:40:36.657245] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.667361] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.677484] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.687610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.697720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.704037] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.704991] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12338953120, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.705023] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.705049] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.705074] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.705084] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.705102] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12338953120, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.705117] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.705133] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.705147] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.705154] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.705168] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.705180] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.705189] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036704968) [2025-05-15 09:40:36.709342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.716957] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.716982] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.717008] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.717030] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.717040] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.717056] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.717066] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.717079] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302036716923, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.717095] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.717105] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.717183] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:36.717197] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:36.717205] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:36.717219] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:36.719472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.722051] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302036721263, register_time_us=-1, curr_time_us=1747302036722049) [2025-05-15 09:40:36.722084] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:36.723281] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.723303] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7548418, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.723317] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7548418, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.723343] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.723360] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.723377] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.723400] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.723419] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036723417, replica_locations:[]}) [2025-05-15 09:40:36.723439] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.723466] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.724751] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.724773] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.724791] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.724816] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.724832] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.724840] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.724857] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.724872] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036724871, replica_locations:[]}) [2025-05-15 09:40:36.724938] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7546744, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=270, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.726641] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.726670] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036726634) [2025-05-15 09:40:36.726690] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302036526590, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.726718] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12338998721, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.726736] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.726762] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.726779] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.726787] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.726804] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.726817] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.726816] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12338826228, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.726824] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036726705) [2025-05-15 09:40:36.726838] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.726857] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.726873] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.726882] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.726897] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.726908] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.726926] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036726803) [2025-05-15 09:40:36.728867] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_mview, table_name.ptr()="data_size:11, data:5F5F616C6C5F6D76696577", ret=-5019) [2025-05-15 09:40:36.728897] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_mview, ret=-5019) [2025-05-15 09:40:36.728912] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_mview, db_name=oceanbase) [2025-05-15 09:40:36.728926] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_mview) [2025-05-15 09:40:36.728941] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:36.728950] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:36.728965] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_mview' doesn't exist [2025-05-15 09:40:36.728974] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=8][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:36.728984] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:36.728994] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:36.729003] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:36.729012] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:36.729022] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:36.729032] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:36.729051] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:36.729061] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.729073] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.729084] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:36.729097] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=12][errcode=-5019] fail to handle text query(stmt=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4, ret=-5019) [2025-05-15 09:40:36.729114] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=14][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:36.729124] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:36.729143] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:36.729162] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.729171] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.729178] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:36.729197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:36.729211] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.729224] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4291][T1_TimerWK2_TntSharedTimer][T1][YB427F000001-000635291D0C63BE-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5019, ret=-5019) [2025-05-15 09:40:36.729236] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:36.729245] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:36.729254] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:36.729264] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302036728600, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:36.729275] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4291][T1_TimerWK2_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:36.729622] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=32][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.733958] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:36.739735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.741705] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3954][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=7] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:36.749848] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.755044] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.755065] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:36.755075] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:36.755095] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:36.755112] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:36.755137] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:36.759963] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.770092] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.780207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.790321] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.800434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.804096] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.805027] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:36.805051] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302036805024, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302036805024, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:36.805090] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.805118] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036805086) [2025-05-15 09:40:36.805134] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302036604994, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.805163] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12339053709, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.805181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.805202] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.805219] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.805227] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.805244] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12339053709, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.805259] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.805274] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.805287] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.805295] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.805309] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.805321] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.805337] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036805148) [2025-05-15 09:40:36.810554] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.810574] WDIAG [SERVER.OMT] check_cgroup_status (ob_cgroup_ctrl.cpp:182) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] check cgroup root dir failed(tmp_ret=-4027) [2025-05-15 09:40:36.816926] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.816949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.816966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.816987] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.817002] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.817018] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.817026] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.817040] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302036816903, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.817055] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.817063] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.817129] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:36.817143] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:36.817151] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:36.817159] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:36.820006] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:36.820682] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.825058] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.825079] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7446638, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.825096] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7446638, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.825145] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.825161] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.825171] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.825187] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.825204] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036825202, replica_locations:[]}) [2025-05-15 09:40:36.825223] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.825249] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.826372] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.826393] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.826410] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.826434] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.826449] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.826463] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.826476] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.826503] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036826502, replica_locations:[]}) [2025-05-15 09:40:36.826562] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7445120, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=271, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.826711] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12339098245, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.826736] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.826761] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.826779] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.826788] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.826805] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.826818] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.826825] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036826696) [2025-05-15 09:40:36.826842] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.826877] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036826837) [2025-05-15 09:40:36.826894] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302036626815, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.826926] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12338925752, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.826943] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.826962] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.826977] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.826985] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.827001] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.827012] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.827018] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036826911) [2025-05-15 09:40:36.827324] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4461][T1001_TimerWK1_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-05-15 09:40:36.827343] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4461][T1001_TimerWK1_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=17] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:36.830797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.832003] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=28] table not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, table_name.ptr()="data_size:25, data:5F5F616C6C5F6469736B5F696F5F63616C6962726174696F6E", ret=-5019) [2025-05-15 09:40:36.832031] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_disk_io_calibration, ret=-5019) [2025-05-15 09:40:36.832043] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_disk_io_calibration, db_name=oceanbase) [2025-05-15 09:40:36.832059] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_disk_io_calibration) [2025-05-15 09:40:36.832098] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=36][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:36.832110] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:36.832126] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_disk_io_calibration' doesn't exist [2025-05-15 09:40:36.832138] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:36.832147] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:36.832158] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:36.832166] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:36.832175] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:36.832188] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:36.832195] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:36.832219] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:36.832232] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.832244] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.832257] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:36.832266] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA", ret=-5019) [2025-05-15 09:40:36.832281] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:36.832294] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA""}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:36.832314] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:36.832335] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.832347] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.832354] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:36.832367] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA""}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:36.832381] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [3955][TimerWK2_ServerGTimer][T1][YB427F000001-000635291F3C6183-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.832395] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [3955][TimerWK2_ServerGTimer][T0][YB427F000001-000635291F3C6183-0-0] [lt=12][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA""}, aret=-5019, ret=-5019) [2025-05-15 09:40:36.832410] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA") [2025-05-15 09:40:36.832419] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:36.832431] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:36.832440] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302036831743, sql=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA") [2025-05-15 09:40:36.832453] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:36.832461] WDIAG [COMMON] parse_calibration_table (ob_io_calibration.cpp:908) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] query failed(ret=-5019, sql_string=select mode, size, latency, iops from __all_disk_io_calibration where svr_ip = "127.0.0.1" and svr_port = 2882 and storage_name = "DATA") [2025-05-15 09:40:36.832540] WDIAG [COMMON] read_from_table (ob_io_calibration.cpp:773) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] parse calibration data failed(ret=-5019) [2025-05-15 09:40:36.832554] WDIAG [SERVER] refresh_io_calibration (ob_server.cpp:3859) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] fail to refresh io calibration from table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.832562] WDIAG [SERVER] runTimerTask (ob_server.cpp:3849) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] ObRefreshIOCalibrationTimeTask task failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.839377] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-05-15 09:40:36.840912] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.851030] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.859912] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:36.859944] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:36.859955] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=9][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:36.859971] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:36.859989] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=14][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:36.860001] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:36.860012] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:36.860024] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:36.860031] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:36.860043] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:36.860050] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:36.860063] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:36.860072] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:36.860083] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:36.860098] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:36.860110] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.860119] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=8][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:36.860130] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:36.860142] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:36.860159] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=16][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:36.860175] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=15][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:36.860204] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=25][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:36.860240] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=33][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.860252] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=12][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:36.860259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:36.860278] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:36.860293] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.860307] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4247][T1_Occam][T1][YB427F000001-000635291C6C6241-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:36.860319] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:36.860333] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:36.860342] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:36.860354] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302036859741, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:36.860369] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:36.860378] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f1abce563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:36.860400] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:36.860484] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1abce563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:36.860509] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:36.860519] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:36.860532] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:36.860552] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:36.861145] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.871265] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.873344] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4624][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=37] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1a903de2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:36.881393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.891502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.897865] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302036897862, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007612, add_bytes: 0 [2025-05-15 09:40:36.898521] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=15] ====== tenant freeze timer task ====== [2025-05-15 09:40:36.901606] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.903747] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=124] PNIO [ratelimit] time: 1747302036903745, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007615, add_bytes: 0 [2025-05-15 09:40:36.904152] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:36.905106] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12339153209, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.905133] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.905158] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.905177] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.905191] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.905206] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12339153209, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:36.905221] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.905237] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:36.905252] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:36.905260] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:36.905273] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.905281] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:36.905293] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036905087) [2025-05-15 09:40:36.911709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.913811] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.916957] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.916979] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.916998] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.917017] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.917032] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.917051] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.917066] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.917075] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302036916936, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.917090] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.917098] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:36.917159] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:36.917172] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:36.917180] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:36.917194] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:36.920062] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=34] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:36.920068] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=26] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:36.921013] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.921814] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.926693] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.926717] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7345006, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.926736] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7345006, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.926765] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.926761] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:36.926783] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.926788] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302036926755) [2025-05-15 09:40:36.926792] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.926804] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302036726705, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:36.926808] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.926823] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036926822, replica_locations:[]}) [2025-05-15 09:40:36.926828] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:36.926844] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:36.926873] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:36.926872] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12339026283, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:36.926895] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.926918] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:36.926853] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:2944224391}, scn={val:1747289697727835002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300530150701, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:1686016, ret_code_:0}}, next_to_submit_lsn_:{lsn:2944224391}, next_to_submit_scn_:{val:1747289697727835002, v:0}, base_lsn_:{lsn:2818400256}, base_scn_:{val:1747274404592930000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69205000, next_round_pread_size:3678208, curr_read_pos:3609237, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3609237, log_storage_:{start_lsn:{lsn:2940615154}, end_lsn:{lsn:2944224391}, read_buf:{buf_len_:3747840, buf_:0x7f1a69205000}, block_size:67104768, log_storage_:0x7f1ab0f4fbf0, log_storage_:{log_tail:{lsn:2944224980}, readable_log_tail:{lsn:2944224980}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:43, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:0, max_block_id:44, curr_writable_block_id:43}, logical_block_size_:67104768, curr_block_writable_size_:8384812, block_header_serialize_buf_:0x7f1ab0f508b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697727835001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697727835001, v:0}, data_checksum:2923900223, flag:13900}}, init_mode_version:0, accumulate_checksum:2881944827, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697727835001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:64, read_io_size:126980096, read_disk_cost_ts:1613509, start_lsn:{lsn:2818400256}}}}}}) [2025-05-15 09:40:36.926930] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.926941] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.926938] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=84] get wrs ts(ls_id={id:1}, delta=12339198776, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:36.926956] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.926955] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:36.926968] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:36.926971] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:36.926975] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036926858) [2025-05-15 09:40:36.926982] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:36.926996] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:36.927007] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.927019] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:36.927026] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302036926821) [2025-05-15 09:40:36.928153] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.928176] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.928194] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.928218] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.928235] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.928248] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.928260] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.928275] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036928274, replica_locations:[]}) [2025-05-15 09:40:36.928339] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=7343342, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=272, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:36.931917] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.934221] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=20] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:36.935074] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4831489288, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:36.935165] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=24] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:36.935181] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4831489288, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:36.936063] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.936085] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.936103] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.936120] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:36.936136] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.936151] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.936161] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.936181] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.936191] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.936207] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:36.936216] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:36.936226] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:36.936246] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:36.936260] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.936275] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.936295] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.936309] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:36.936318] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:36.936331] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:36.936341] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=6] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302036936340, replica_locations:[]}) [2025-05-15 09:40:36.936361] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=1][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:36.936413] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:36.936431] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:36.936468] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=36] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:36.936504] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=37][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:36.936529] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=24] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=133, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:36.936553] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=24][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:36.936578] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=23][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:36.936586] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:36.942026] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.952134] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.954737] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] [COMPACTION DAG STATUS] (start_time=1747302026953891, end_time=1747302036954734, dag_status={}) [2025-05-15 09:40:36.954764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=25] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954784] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=19] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954792] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954804] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954813] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954825] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954834] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954848] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=13] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954855] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=7] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:36.954867] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=8] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-15 09:40:36.954879] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [4570][T1002_DagSchedu][T1002][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-15 09:40:36.958949] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:36.958977] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:36.958987] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:36.959007] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=17] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:36.959025] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11799849374, flush_io_finished_ret_:0, flush_io_finished_round_:1511, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:36.959654] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:36.962238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.972345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.977832] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:49, tid:3961}, {errcode:-4721, dropped:411, tid:4291}]) [2025-05-15 09:40:36.982452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.992571] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:36.995668] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE0-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.995688] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE0-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.995707] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE0-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.995745] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE0-0-0] [lt=36][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.995760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE0-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.995773] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:36.995788] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.995799] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.995812] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302036995646, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:36.995828] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.995837] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:36.995910] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:36.997144] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE1-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.997161] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE1-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:36.997175] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE1-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:36.997193] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE1-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:36.997201] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE1-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:36.997215] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:36.997223] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:36.997235] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:36.997242] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302036997135, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:36.997256] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:36.997263] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:36.997312] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:36.998682] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:36.999279] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:36.999303] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:36.999314] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:36.999330] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:36.999344] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:36.999357] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:36.999371] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:36.999380] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:36.999374] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.999398] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:36.999414] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.999428] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:36.999436] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:36.999451] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:36.999471] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:36.999498] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:36.999541] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:37.002679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.004204] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.005148] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.005178] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037005141) [2025-05-15 09:40:37.005197] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302036805148, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.005239] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12339253754, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.005259] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.005283] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.005306] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.005314] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.005327] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1001}, delta=12339253754, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.005337] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.005348] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.005357] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.005365] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.005374] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.005384] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.005391] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037005217) [2025-05-15 09:40:37.012790] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.015668] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=61] tx gc loop thread is running(MTL_ID()=1002) [2025-05-15 09:40:37.015703] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=35] try gc retain ctx [2025-05-15 09:40:37.015712] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=8] try to retry start_working [2025-05-15 09:40:37.015752] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289693216794002, v:0}, keep_alive_scn_:{val:1747289693216794002, v:0}, update_ts_:1747302037015521}) [2025-05-15 09:40:37.016337] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=25] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:37.016431] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=81] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:0, v:0}, keep_alive_scn_:{val:0, v:0}, update_ts_:0}) [2025-05-15 09:40:37.017018] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:37.017121] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.017145] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.017168] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.017185] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.017197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.017209] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.017221] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.017233] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.017245] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302037017094, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.017258] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.017267] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.017342] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.017353] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.017364] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.017375] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.022219] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.022243] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.022259] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.022896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.023348] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:37.023368] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.023384] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.023400] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:37.023411] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:37.023424] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:37.026818] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:37.026871] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40] get wrs ts(ls_id={id:1}, delta=12339298250, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.026887] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.026905] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.026920] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.026929] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.026920] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.026943] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.026943] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037026916) [2025-05-15 09:40:37.026952] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.026955] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302036826908, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.026959] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037026858) [2025-05-15 09:40:37.026975] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:37.026998] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] get_min_unreplayed_log_info(lsn={lsn:5311387238}, scn={val:1747289697900328001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300529803179, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:5311387238}, next_to_submit_scn_:{val:1747289697900328001, v:0}, base_lsn_:{lsn:5167067136}, base_scn_:{val:1747274408663046000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a72405000, next_round_pread_size:3678208, curr_read_pos:2755105, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2755105, log_storage_:{start_lsn:{lsn:5308632133}, end_lsn:{lsn:5311387238}, read_buf:{buf_len_:3747840, buf_:0x7f1a72405000}, block_size:67104768, log_storage_:0x7f1a97bcbbf0, log_storage_:{log_tail:{lsn:5311387831}, readable_log_tail:{lsn:5311387831}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:79, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:133, min_block_id:61, max_block_id:80, curr_writable_block_id:79}, logical_block_size_:67104768, curr_block_writable_size_:56993609, block_header_serialize_buf_:0x7f1a97bcc8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697900328000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:431, scn_:{val:1747289697900328000, v:0}, data_checksum:4152786848, flag:25505}}, init_mode_version:0, accumulate_checksum:3961629555, curr_entry_is_padding:0, padding_entry_size:463, padding_entry_scn:{val:1747289697900328000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:76, read_io_size:145543168, read_disk_cost_ts:2159331, start_lsn:{lsn:5167067136}}}}}}) [2025-05-15 09:40:37.027084] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=86] get wrs ts(ls_id={id:1}, delta=12339125757, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.027097] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.027113] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.027126] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.027135] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.027147] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.027156] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.027164] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037026968) [2025-05-15 09:40:37.027769] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4606][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=27] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:37.028466] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.028485] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7243231, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.028513] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7243231, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.028528] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.028540] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=7243231, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.028556] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.028574] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.028590] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.028604] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.028615] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.028631] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.028649] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037028648, replica_locations:[]}) [2025-05-15 09:40:37.028667] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.028686] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:272, local_retry_times:272, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:37.028706] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.028718] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:37.028733] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:37.028744] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:37.028750] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:37.028766] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:37.028778] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.028825] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11800919302, cache_obj->added_lc()=false, cache_obj->get_object_id()=15795, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.029992] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.030020] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.030055] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.030069] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.030084] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.030094] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.030105] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.030118] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.030134] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.030149] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.030168] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.030184] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.030199] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.030213] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037030212, replica_locations:[]}) [2025-05-15 09:40:37.030236] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.030250] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.030263] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.030280] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:37.030288] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:37.030300] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:37.030319] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:37.030334] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:37.030348] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:37.030360] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:37.030371] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:37.030381] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:37.030394] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:37.030406] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:37.030417] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:37.030426] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:37.030436] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:37.030447] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:37.030456] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:37.030472] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:37.030484] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.030506] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:37.030518] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:37.030531] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:37.030541] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=273, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:37.030567] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] will sleep(sleep_us=100000, remain_us=7241115, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=273, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.033006] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.034098] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] LogLoopThread round_cost_time(us)(round_cost_time=25) [2025-05-15 09:40:37.034125] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:37.034138] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.034152] WDIAG [PALF] get_leader_from_cache_ (log_sliding_window.cpp:2149) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock_get_leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:37.034164] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_fetch_log_dst failed or invalid(ret=-4721, palf_id=1, self="127.0.0.1:2882", fetch_log_dst="0.0.0.0:0") [2025-05-15 09:40:37.034176] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.039471] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=5] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-05-15 09:40:37.043118] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.053226] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.062220] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:37.062263] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:37.063337] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.064412] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.064433] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.064479] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:37.064500] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:37.064517] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:37.064505] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC969A-0-0] [lt=27][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302037064465}) [2025-05-15 09:40:37.064532] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:37.064542] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.064564] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:37.064575] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:37.064585] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:37.064594] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:37.064603] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:37.064612] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:37.064623] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:37.064964] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:37.064977] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.064994] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.065003] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:37.065011] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:37.065019] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:37.065027] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:37.065037] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.065051] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:37.069517] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:37.069545] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.069555] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:37.069565] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:37.069572] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:37.069649] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=77][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:37.073447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.076539] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553050, support_ssl=false) [2025-05-15 09:40:37.076635] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221553050, sess.client_addr_="172.22.0.1:46364") [2025-05-15 09:40:37.076655] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f620db0, session_id:3221553050, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.076903] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.076939] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.076964] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.076987] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.077002] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.077010] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.077037] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.077055] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553050, proxy_sessid=0) [2025-05-15 09:40:37.077091] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=13][errcode=0] server close connection(sessid=3221553050, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.077103] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553050, proxy_sessid=0) [2025-05-15 09:40:37.077111] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.077120] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9085-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553050, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.077170] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f620db0, session_id:3221553050, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302037075848, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.077193] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553050, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.077205] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a6f620db0, session_id:3221553050, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302037075848, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.077224] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553050) [2025-05-15 09:40:37.077240] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553050, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.083566] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.093677] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.103783] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.104265] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.105227] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12339353219, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.105258] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.105281] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.105299] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.105326] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.105344] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12339353219, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.105358] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.105373] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.105384] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.105393] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.105406] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.105415] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.105425] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037105201) [2025-05-15 09:40:37.113892] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.115397] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:37.115418] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:37.115426] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=6] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:37.115443] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=17] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:37.117162] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.117182] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.117204] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.117221] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.117231] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.117243] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.117256] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.117268] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.117281] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302037117133, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.117293] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.117303] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.117379] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.117390] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.117403] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.117414] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.119132] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553052, support_ssl=false) [2025-05-15 09:40:37.119206] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553052, sess.client_addr_="172.22.0.1:46374") [2025-05-15 09:40:37.119211] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553053, support_ssl=false) [2025-05-15 09:40:37.119224] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553052, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.119298] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221553053, sess.client_addr_="172.22.0.1:46388") [2025-05-15 09:40:37.119317] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de2adb0, session_id:3221553053, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.119480] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.119532] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=29] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.119530] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.119566] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.119563] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.119585] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.119590] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221553054, support_ssl=false) [2025-05-15 09:40:37.119602] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.119610] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.119603] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.119629] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.119635] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553054, sess.client_addr_="172.22.0.1:46398") [2025-05-15 09:40:37.119638] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.119642] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.119655] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.119657] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553052, proxy_sessid=0) [2025-05-15 09:40:37.119648] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553054, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.119676] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.119684] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=8][errcode=0] server close connection(sessid=3221553052, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.119697] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553052, proxy_sessid=0) [2025-05-15 09:40:37.119698] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553053, proxy_sessid=0) [2025-05-15 09:40:37.119710] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:37.119719] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9074-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553052, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.119740] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=12][errcode=0] server close connection(sessid=3221553053, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.119751] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553053, proxy_sessid=0) [2025-05-15 09:40:37.119761] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:37.119773] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553052, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037119225, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.119774] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B6-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553053, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.119796] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553052, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.119807] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553052, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037119225, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.119824] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553052) [2025-05-15 09:40:37.119827] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2adb0, session_id:3221553053, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037119225, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.119852] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553053, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.119841] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553052, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.119861] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de2adb0, session_id:3221553053, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037119225, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.119880] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553053) [2025-05-15 09:40:37.119885] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.119898] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553053, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.119911] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.119943] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.119964] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.119979] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.119986] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.120008] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.120022] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553054, proxy_sessid=0) [2025-05-15 09:40:37.120051] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=13][errcode=0] server close connection(sessid=3221553054, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.120064] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553054, proxy_sessid=0) [2025-05-15 09:40:37.120070] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:37.120078] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9086-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553054, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.120125] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553054, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302037119225, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.120148] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553054, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.120155] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553054, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302037119225, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.120170] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553054) [2025-05-15 09:40:37.120181] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553054, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.120223] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553055, support_ssl=false) [2025-05-15 09:40:37.120238] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:37.120266] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553055, sess.client_addr_="172.22.0.1:46408") [2025-05-15 09:40:37.120281] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553055, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.120550] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.120587] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.120602] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.120624] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.120633] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.120646] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.120660] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.120635] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553056, support_ssl=false) [2025-05-15 09:40:37.120681] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221553055, proxy_sessid=0) [2025-05-15 09:40:37.120703] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=10][errcode=0] server close connection(sessid=3221553055, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.120715] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553055, proxy_sessid=0) [2025-05-15 09:40:37.120722] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.120725] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=57] sm conn init succ(conn.sessid_=3221553056, sess.client_addr_="172.22.0.1:46418") [2025-05-15 09:40:37.120736] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9075-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553055, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.120746] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553056, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.120795] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=37] new mysql sessid created(conn.sessid_=3221553057, support_ssl=false) [2025-05-15 09:40:37.120794] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553055, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.120821] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553055, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.120831] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221553057, sess.client_addr_="172.22.0.1:46416") [2025-05-15 09:40:37.120836] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553055, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.120845] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553057, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.120854] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553055) [2025-05-15 09:40:37.120874] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221553055, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.120948] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221553058, support_ssl=false) [2025-05-15 09:40:37.120983] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553059, support_ssl=false) [2025-05-15 09:40:37.121016] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221553058, sess.client_addr_="172.22.0.1:46438") [2025-05-15 09:40:37.121018] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553059, sess.client_addr_="172.22.0.1:46446") [2025-05-15 09:40:37.121037] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553059, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121035] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f620030, session_id:3221553058, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121086] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.121144] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.121156] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.121171] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.121183] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.121192] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.121209] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121221] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553056, proxy_sessid=0) [2025-05-15 09:40:37.121243] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=10][errcode=0] server close connection(sessid=3221553056, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121256] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553056, proxy_sessid=0) [2025-05-15 09:40:37.121255] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.121267] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:37.121167] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.121274] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B7-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553056, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.121292] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.121304] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.121315] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221553060, support_ssl=false) [2025-05-15 09:40:37.121320] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.121333] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.121330] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=126] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.121344] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.121344] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.121356] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.121357] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121357] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553060, sess.client_addr_="172.22.0.1:46478") [2025-05-15 09:40:37.121364] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.121370] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553058, proxy_sessid=0) [2025-05-15 09:40:37.121375] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.121373] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553060, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121391] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=11][errcode=0] server close connection(sessid=3221553058, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121392] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121400] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553061, support_ssl=false) [2025-05-15 09:40:37.121404] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553058, proxy_sessid=0) [2025-05-15 09:40:37.121412] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.121408] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553057, proxy_sessid=0) [2025-05-15 09:40:37.121419] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9087-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553058, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.121427] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553061, sess.client_addr_="172.22.0.1:46490") [2025-05-15 09:40:37.121430] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=12][errcode=0] server close connection(sessid=3221553057, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121413] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553056, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121441] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553057, proxy_sessid=0) [2025-05-15 09:40:37.121440] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de2adb0, session_id:3221553061, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121449] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:37.121454] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] kill and revert session(conn.sessid_=3221553056, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.121456] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9076-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553057, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.121475] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553056, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.121482] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f620030, session_id:3221553058, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121505] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] mark session id unused(conn.sessid_=3221553056) [2025-05-15 09:40:37.121520] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221553058, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.121530] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f620030, session_id:3221553058, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.121550] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553058) [2025-05-15 09:40:37.121532] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553056, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.121552] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.121567] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553058, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.121584] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.121588] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553057, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121600] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.121615] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221553057, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.121617] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.121629] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.121638] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.121652] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121662] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553060, proxy_sessid=0) [2025-05-15 09:40:37.121665] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.121681] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=9][errcode=0] server close connection(sessid=3221553060, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121679] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553057, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037120283, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.121687] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.121692] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553060, proxy_sessid=0) [2025-05-15 09:40:37.121702] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:37.121700] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553057) [2025-05-15 09:40:37.121702] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.121715] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.121710] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B8-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553060, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.121727] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.121717] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553057, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.121734] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.121747] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121761] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553061, proxy_sessid=0) [2025-05-15 09:40:37.121769] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553062, support_ssl=false) [2025-05-15 09:40:37.121779] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=7][errcode=0] server close connection(sessid=3221553061, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121791] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553061, proxy_sessid=0) [2025-05-15 09:40:37.121798] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.121809] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9088-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553061, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.121815] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553062, sess.client_addr_="172.22.0.1:46462") [2025-05-15 09:40:37.121803] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553060, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121834] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553062, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121864] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2adb0, session_id:3221553061, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.121873] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.121886] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553060, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.121895] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] kill and revert session(conn.sessid_=3221553061, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.121903] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.121908] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553060, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.121918] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.121926] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553060) [2025-05-15 09:40:37.121930] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.121941] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.121945] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221553060, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.121955] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.121965] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a1de2adb0, session_id:3221553061, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.121983] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553061) [2025-05-15 09:40:37.121973] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.121993] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553059, proxy_sessid=0) [2025-05-15 09:40:37.121992] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553061, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.122025] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=13][errcode=0] server close connection(sessid=3221553059, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.122029] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.122040] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553059, proxy_sessid=0) [2025-05-15 09:40:37.122049] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:37.122058] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.122056] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9077-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553059, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.122073] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.122089] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.122099] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.122108] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.122108] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553059, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.122125] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.122134] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553059, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.122136] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553062, proxy_sessid=0) [2025-05-15 09:40:37.122150] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553059, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.122160] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=12][errcode=0] server close connection(sessid=3221553062, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.122170] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553062, proxy_sessid=0) [2025-05-15 09:40:37.122170] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553059) [2025-05-15 09:40:37.122176] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:37.122184] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90B9-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553062, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.122184] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553059, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.122219] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553062, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.122240] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553062, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.122287] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553062, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037121341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.122283] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.122308] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553062) [2025-05-15 09:40:37.122309] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.122325] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.122320] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553062, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.123424] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:37.123475] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.123507] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.123527] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:37.123540] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:37.123548] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:37.124013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.126877] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.126917] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037126870) [2025-05-15 09:40:37.126932] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302036926820, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.126962] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12339398815, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.126977] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.126980] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.126996] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037126973) [2025-05-15 09:40:37.127001] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.127008] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302037026968, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.127017] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.127025] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.127031] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12339226322, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.127040] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.127052] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.127046] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.127061] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037126944) [2025-05-15 09:40:37.127061] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.127072] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.127082] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.127095] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.127104] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.127112] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037127018) [2025-05-15 09:40:37.130735] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.130765] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7140978, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.130786] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7140978, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.130803] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.130822] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=7140978, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.130839] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.130858] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.130879] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.130896] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.130909] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.130927] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.130947] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037130944, replica_locations:[]}) [2025-05-15 09:40:37.130967] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.130989] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:273, local_retry_times:273, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:37.131015] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.131028] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:37.131045] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:37.131056] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:37.131067] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:37.131104] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:37.131119] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.131185] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11801021658, cache_obj->added_lc()=false, cache_obj->get_object_id()=15796, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.132609] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.132639] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.132675] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.132690] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.132706] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.132717] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.132728] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.132742] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.132757] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.132771] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.132785] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.132804] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.132819] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.132834] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037132833, replica_locations:[]}) [2025-05-15 09:40:37.132851] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.132864] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.132877] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.132893] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:37.132901] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:37.132913] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:37.132930] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:37.132946] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:37.132958] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:37.132968] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:37.132979] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:37.132989] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:37.133003] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:37.133015] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:37.133027] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:37.133038] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:37.133049] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:37.133061] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:37.133070] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:37.133087] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:37.133100] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.133111] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:37.133122] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:37.133137] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:37.133147] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=274, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:37.133175] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=7138507, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=274, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.134127] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.134186] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:37.134209] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.134222] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.134486] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:37.144239] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.154349] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.160476] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=16] table not exist(tenant_id=1, database_id=201001, table_name=__all_unit, table_name.ptr()="data_size:10, data:5F5F616C6C5F756E6974", ret=-5019) [2025-05-15 09:40:37.160522] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=42][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_unit, ret=-5019) [2025-05-15 09:40:37.160621] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=97][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_unit, db_name=oceanbase) [2025-05-15 09:40:37.160636] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_unit) [2025-05-15 09:40:37.160650] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:37.160660] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:37.160677] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_unit' doesn't exist [2025-05-15 09:40:37.160687] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:37.160697] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:37.160706] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:37.160716] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:37.160726] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:37.160736] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:37.160745] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:37.160767] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:37.160777] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.160789] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.160800] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:37.160810] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2025-05-15 09:40:37.160823] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:37.160833] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:37.160852] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:37.160873] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:37.160882] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:37.160890] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:37.160910] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:37.160924] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.160936] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)"}, aret=-5019, ret=-5019) [2025-05-15 09:40:37.160946] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2025-05-15 09:40:37.160957] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:37.160967] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:37.160977] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302037160210, sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2025-05-15 09:40:37.160990] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:37.160999] WDIAG [SHARE] read_units (ob_unit_table_operator.cpp:1238) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] execute sql failed(sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1), ret=-5019) [2025-05-15 09:40:37.161087] WDIAG [SHARE] get_units_by_tenant (ob_unit_table_operator.cpp:883) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] read_units failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * from __all_unit where resource_pool_id in (select resource_pool_id from __all_resource_pool where tenant_id = 1)) [2025-05-15 09:40:37.161099] WDIAG [SHARE] get_sys_unit_count (ob_unit_table_operator.cpp:62) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=11][errcode=-5019] failed to get units by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:37.161109] WDIAG [SHARE] get_sys_unit_count (ob_unit_getter.cpp:453) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] ut_operator get sys unit count failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:37.161118] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:84) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] get sys unit count fail(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:37.161128] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:98) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9] refresh tenant units(sys_unit_cnt=0, units=[], ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:37.162392] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, table_name.ptr()="data_size:12, data:5F5F616C6C5F74656E616E74", ret=-5019) [2025-05-15 09:40:37.162412] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=19][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_tenant, ret=-5019) [2025-05-15 09:40:37.162424] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_tenant, db_name=oceanbase) [2025-05-15 09:40:37.162436] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_tenant) [2025-05-15 09:40:37.162446] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:37.162455] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:37.162466] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_tenant' doesn't exist [2025-05-15 09:40:37.162472] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:37.162480] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:37.162495] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=15][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:37.162504] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:37.162512] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:37.162520] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:37.162527] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:37.162540] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:37.162548] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.162557] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.162566] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:37.162574] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT tenant_id FROM __all_tenant, ret=-5019) [2025-05-15 09:40:37.162583] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:37.162591] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:37.162604] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=11][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:37.162616] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:37.162625] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:37.162630] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=5][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:37.162641] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=7][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:37.162652] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4196][OmtNodeBalancer][T1][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.162661] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT tenant_id FROM __all_tenant"}, aret=-5019, ret=-5019) [2025-05-15 09:40:37.162671] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT tenant_id FROM __all_tenant) [2025-05-15 09:40:37.162680] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:37.162689] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:37.162704] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=15][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302037162331, sql=SELECT tenant_id FROM __all_tenant) [2025-05-15 09:40:37.162714] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:37.162723] WDIAG [SHARE] read_tenants (ob_unit_table_operator.cpp:1301) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] execute sql failed(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2025-05-15 09:40:37.162767] WDIAG [SHARE] get_tenants (ob_unit_table_operator.cpp:103) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] fail read tenants(sql=SELECT tenant_id FROM __all_tenant, ret=-5019) [2025-05-15 09:40:37.162777] WDIAG [SHARE] get_tenants (ob_unit_getter.cpp:209) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=10][errcode=-5019] ut_operator get_resource_pools failed(ret=-5019) [2025-05-15 09:40:37.162785] WDIAG [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:114) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=8][errcode=-5019] get cluster tenants fail(ret=-5019) [2025-05-15 09:40:37.162827] INFO [SERVER] cal_all_part_disk_default_percentage (ob_server_utils.cpp:309) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=7] cal_all_part_disk_default_percentage succ(data_dir="/root/ob/observer/store/sstable", clog_dir="/root/ob/observer/store/clog", shared_mode=true, data_disk_total_size=3736332664832, data_disk_default_percentage=60, clog_disk_total_size=3736332664832, clog_disk_default_percentage=30) [2025-05-15 09:40:37.162843] INFO [SERVER] decide_disk_size (ob_server_utils.cpp:345) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=14] decide disk size finished(suggested_disk_size=21474836480, suggested_disk_percentage=0, default_disk_percentage=30, total_space=3736332664832, disk_size=21474836480) [2025-05-15 09:40:37.162853] INFO [SERVER] get_log_disk_info_in_config (ob_server_utils.cpp:88) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=9] get_log_disk_info_in_config(suggested_data_disk_size=21474836480, suggested_clog_disk_size=21474836480, suggested_data_disk_percentage=0, suggested_clog_disk_percentage=0, log_disk_size=21474836480, log_disk_percentage=0, total_log_disk_size=3736332664832) [2025-05-15 09:40:37.162867] INFO [CLOG] try_resize (ob_server_log_block_mgr.cpp:782) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=11] try_resize success(ret=0, log_disk_size=21474836480, total_log_disk_size=3736332664832, this={dir::"/root/ob/observer/store/clog/log_pool", dir_fd:16, meta_fd:17, log_pool_meta:{curr_total_size:21474836480, next_total_size:21474836480, status:0}, min_block_id:154, max_block_id:381, min_log_disk_size_for_all_tenants_:21474836480, is_inited:true}) [2025-05-15 09:40:37.162886] INFO [SERVER.OMT] run1 (ob_tenant_node_balancer.cpp:122) [4196][OmtNodeBalancer][T0][YB427F000001-000635291A8C61AA-0-0] [lt=18] refresh tenant config(tenants=[], ret=-5019) [2025-05-15 09:40:37.164458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.173245] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4292][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-05-15 09:40:37.173271] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4292][T1_TimerWK3_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=20] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:37.174259] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.174506] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.174578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.174725] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.174729] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.184689] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.191403] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=1] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:37.191429] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:37.191443] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:37.191456] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:37.191465] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:37.191473] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:37.191484] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:37.191502] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=17][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:37.191510] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:37.191562] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6443-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.191693] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:37.194787] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.204324] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.204899] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.205266] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.205298] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037205259) [2025-05-15 09:40:37.205315] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302037005216, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.205340] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:37.205368] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] get_min_unreplayed_log_info(lsn={lsn:934287681}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302037120520, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:934287681}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:872361984}, base_scn_:{val:1747274406063927000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69605000, next_round_pread_size:3678208, curr_read_pos:3075493, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3075493, log_storage_:{start_lsn:{lsn:931212188}, end_lsn:{lsn:934287681}, read_buf:{buf_len_:3747840, buf_:0x7f1a69605000}, block_size:67104768, log_storage_:0x7f1a73fd1bf0, log_storage_:{log_tail:{lsn:934288498}, readable_log_tail:{lsn:934288498}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:13, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:143, min_block_id:0, max_block_id:14, curr_writable_block_id:13}, logical_block_size_:67104768, curr_block_writable_size_:5178254, block_header_serialize_buf_:0x7f1a73fd28b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:2923900223, flag:16486}}, init_mode_version:0, accumulate_checksum:910368143, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}}}}) [2025-05-15 09:40:37.205459] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=87] get wrs ts(ls_id={id:1}, delta=12339453740, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.205477] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.205510] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.205525] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.205536] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.205553] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12339453740, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.205565] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.205579] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.205590] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.205599] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.205611] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.205621] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.205630] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037205328) [2025-05-15 09:40:37.215005] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.217092] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.217115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.217134] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.217147] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.217158] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.217167] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.217178] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.217190] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.217200] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302037217072, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.217212] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.217220] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.217274] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.217284] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.217295] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.217306] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.222341] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.222366] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.222383] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.223476] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:37.223509] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.223521] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.223538] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:37.223549] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:37.223561] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:37.223742] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4291][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=24] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:37.225115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.226945] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12339498278, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.226967] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.226993] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.227008] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.227035] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.227055] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037227030) [2025-05-15 09:40:37.227069] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302037127018, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.227089] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12339325785, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.227101] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.227115] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.233308] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.233330] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7038390, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.233397] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=67][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=7038390, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.233412] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.233421] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=7038390, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.233436] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.233450] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.233467] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.233483] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.233508] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.233524] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.233550] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037233548, replica_locations:[]}) [2025-05-15 09:40:37.233568] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.233588] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:274, local_retry_times:274, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:37.233612] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.233624] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:37.233643] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:37.233653] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:37.233662] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:37.233682] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:37.233696] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.233747] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11801124225, cache_obj->added_lc()=false, cache_obj->get_object_id()=15797, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.234245] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:37.234266] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:37.234279] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:37.234966] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.234996] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.235033] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.235055] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.235071] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.235085] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.235096] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.235110] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.235122] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.235138] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.235152] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.235164] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.235180] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.235194] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037235193, replica_locations:[]}) [2025-05-15 09:40:37.235210] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.235224] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.235225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.235237] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:37.235253] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:37.235265] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:37.235276] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:37.235294] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:37.235308] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:37.235319] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:37.235330] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:37.235341] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:37.235351] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:37.235366] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:37.235378] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:37.235386] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:37.235398] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:37.235405] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:37.235416] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:37.235427] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:37.235444] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:37.235455] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.235464] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:37.235476] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:37.235499] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:37.235511] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=275, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:37.235536] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=7036145, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=275, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.242458] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=27] Cache replace map node details(ret=0, replace_node_count=0, replace_time=2889, replace_start_pos=6000000, replace_num=100000) [2025-05-15 09:40:37.242480] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=10) [2025-05-15 09:40:37.245344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.255178] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.255196] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:37.255205] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:37.255222] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:37.255234] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=10398720) [2025-05-15 09:40:37.255258] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=24, size_used=0, mem_used=0) [2025-05-15 09:40:37.255284] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:37.255829] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.259114] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:37.259143] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:37.259156] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:37.259175] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:37.259191] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=16] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11800149535, flush_io_finished_ret_:0, flush_io_finished_round_:1511, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:37.266472] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.268228] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=29] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:37.274412] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=44][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:37.274434] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=22][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.274450] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=9][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.274470] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.274482] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.274504] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=17][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.274592] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=88][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.274612] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.274632] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.274650] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.274660] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.274678] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.274708] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=12] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.274725] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=13] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:37.276483] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=18] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:37.276534] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=47][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:37.276549] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:37.276563] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:37.276578] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:37.276589] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:37.276599] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=33][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.276608] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=13][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:37.276619] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:37.276629] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:37.276640] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=10][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:37.276651] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:37.276664] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:37.276675] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:37.276684] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:37.276705] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:37.276716] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.276731] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.276741] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=8][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:37.276783] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4291][T1_TimerWK2_LSMetaCh][T1][YB427F000001-000635291D0C63BF-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.286705] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.296814] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.297759] WDIAG load_file_to_string (utility.h:673) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:37.297780] WDIAG get_ethernet_speed (utility.cpp:575) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:37.297792] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:37.297804] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:37.304389] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.305344] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12339553231, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.305372] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.305397] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.305434] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1001}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:37.305455] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:747348431}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302037220757, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:747348431}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:738152448}, base_scn_:{val:1747289697751194001, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a5a205000, next_round_pread_size:3678208, curr_read_pos:1839725, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1839725, log_storage_:{start_lsn:{lsn:745508706}, end_lsn:{lsn:747348431}, read_buf:{buf_len_:3747840, buf_:0x7f1a5a205000}, block_size:67104768, log_storage_:0x7f1a73f61bf0, log_storage_:{log_tail:{lsn:747348553}, readable_log_tail:{lsn:747348553}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:11, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:140, min_block_id:0, max_block_id:12, curr_writable_block_id:11}, logical_block_size_:67104768, curr_block_writable_size_:57908663, block_header_serialize_buf_:0x7f1a73f628b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:154833137, flag:60041}}, init_mode_version:0, accumulate_checksum:2356512705, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}}}}) [2025-05-15 09:40:37.305551] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=95] get wrs ts(ls_id={id:1001}, delta=12339554288, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.305561] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.306919] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.317027] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.317256] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.317300] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=42][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.317336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.317358] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.317370] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.317380] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.322399] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:37.322427] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.323540] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.326996] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:37.327018] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.327041] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037326990) [2025-05-15 09:40:37.327056] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302037126944, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.327087] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12339598821, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.327092] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.327110] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.327123] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037327087) [2025-05-15 09:40:37.327136] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302037227078, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.327137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.327160] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12339426328, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.327175] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.334304] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.334779] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=37] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:37.335018] INFO [COMMON] wash (ob_kvcache_store.cpp:355) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash time detail, (compute_wash_size_time=94, refresh_score_time=39, wash_time=0) [2025-05-15 09:40:37.335687] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.335713] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6936019, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.335731] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6936019, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.335746] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.335759] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=6936019, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.335775] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.335789] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.335809] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.335825] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.335837] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.335854] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.335872] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037335870, replica_locations:[]}) [2025-05-15 09:40:37.335891] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.335911] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.335934] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.335947] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:37.335981] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.336045] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11801226518, cache_obj->added_lc()=false, cache_obj->get_object_id()=15798, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.337434] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.337457] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.337473] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.337501] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.337509] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.337524] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.337541] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.337556] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.337571] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.337584] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.337601] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.337616] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037337615, replica_locations:[]}) [2025-05-15 09:40:37.337688] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6933994, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=276, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.337244] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13] dump tenant info(tenant={id:1, tenant_meta:{unit:{tenant_id:1, unit_id:1, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1, name:"sys_unit_config", resource:{min_cpu:3, max_cpu:3, memory_size:"2GB", log_disk_size:"5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:3, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:3, }}, mode:0, create_timestamp:1746497860185946, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", total_worker_cnt:40, min_worker_cnt:32, max_worker_cnt:150, stopped:0, worker_us:148797700, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:32, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300520045442, tenant_role:1}) [2025-05-15 09:40:37.337803] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=564] dump tenant info(tenant={id:508, tenant_meta:{unit:{tenant_id:508, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:5, max_cpu:5, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:5, }}, mode:0, create_timestamp:1747300516548130, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:508, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"5.000000000000000000e+00", unit_max_cpu:"5.000000000000000000e+00", total_worker_cnt:60, min_worker_cnt:52, max_worker_cnt:150, stopped:0, worker_us:18502030, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:52, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300516557108, tenant_role:0}) [2025-05-15 09:40:37.338268] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=470] dump tenant info(tenant={id:509, tenant_meta:{unit:{tenant_id:509, unit_id:1000, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1000, name:"virtual_tenant_unit", resource:{min_cpu:2.5, max_cpu:2.5, memory_size:"1GB", log_disk_size:"2GB", data_disk_size:0, min_iops:10000, max_iops:50000, iops_weight:0, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:2, }}, mode:0, create_timestamp:1747300516541652, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:509, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:true, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"2.500000000000000000e+00", unit_max_cpu:"2.500000000000000000e+00", total_worker_cnt:35, min_worker_cnt:27, max_worker_cnt:150, stopped:0, worker_us:11963968, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:27, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300516564847, tenant_role:0}) [2025-05-15 09:40:37.338749] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=486] dump tenant info(tenant={id:1001, tenant_meta:{unit:{tenant_id:1001, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"singi_unit", resource:{min_cpu:1.3, max_cpu:1.3, memory_size:"4.02266846GB", log_disk_size:"1.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:13, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:13, }}, mode:0, create_timestamp:1746497897813453, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1001, replay_start_point:ObLogCursor{file_id=1, log_id=1, offset=0}, ls_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.300000000000000044e+00", unit_max_cpu:"1.300000000000000044e+00", total_worker_cnt:15, min_worker_cnt:7, max_worker_cnt:150, stopped:0, worker_us:46568320, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:7, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300522736765, tenant_role:1}) [2025-05-15 09:40:37.339217] INFO [SERVER.OMT] run1 (ob_multi_tenant.cpp:2621) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=473] dump tenant info(tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1001, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"singi_unit", resource:{min_cpu:13, max_cpu:13, memory_size:"36.2040161GB", log_disk_size:"13.5GB", data_disk_size:0, min_iops:9223372036854775807, max_iops:9223372036854775807, iops_weight:13, max_net_bandwidth:INT64_MAX, net_bandwidth_weight:13, }}, mode:0, create_timestamp:1746497897813453, is_removed:false, hidden_sys_data_disk_config_size:0}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=8, log_id=1304693, offset=56638805}, ls_meta_entry:{[ver=1,mode=0,seq=619756][2nd=8512]}, tablet_meta_entry:{[ver=1,mode=0,seq=0][2nd=18446744073709551615]}, is_hidden:false, version:4, snapshot_cnt:0, preallocated_seqs:{object_seq:60000, tmp_file_seq:60000, write_seq:60000}, auto_inc_ls_epoch:0, ls_cnt:0}, create_status:1, epoch:0}, unit_min_cpu:"1.300000000000000000e+01", unit_max_cpu:"1.300000000000000000e+01", total_worker_cnt:62, min_worker_cnt:54, max_worker_cnt:463, stopped:0, worker_us:216776417, recv_hp_rpc_cnt:0, recv_np_rpc_cnt:0, recv_lp_rpc_cnt:0, recv_mysql_cnt:0, recv_task_cnt:0, recv_large_req_cnt:0, tt_large_quries:0, workers:54, nesting workers:8, req_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 , multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=0 cnt[3]=0 cnt[4]=0 cnt[5]=0 cnt[6]=0 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:, rpc_stat_info:, token_change_ts:1747300525931346, tenant_role:0}) [2025-05-15 09:40:37.339749] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=526][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.349881] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.354601] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:37.354635] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.354651] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:37.354665] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:37.359991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.363733] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:37.363804] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:37.363871] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:37.363939] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:37.363968] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:16402, total:16412) [2025-05-15 09:40:37.364002] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:37.364057] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=49] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:16402, total:16412) [2025-05-15 09:40:37.364086] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:37.364106] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:37.364121] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:37.364151] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] successfully alloc block(macro_id={[ver=1,mode=0,seq=628772][2nd=1899]}) [2025-05-15 09:40:37.370105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.374224] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4501][T1001_ArcSender][T1001][YB427F000001-000635291CBC6112-0-0] [lt=26][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:37.378634] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:32, tid:4630}]) [2025-05-15 09:40:37.380215] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.390324] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.395543] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=33] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.395756] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.395851] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=129] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.395877] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.395906] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.396877] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=38] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.396983] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:37.400434] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.404436] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.405379] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.405422] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037405373) [2025-05-15 09:40:37.405438] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302037205327, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.405502] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] get wrs ts(ls_id={id:1}, delta=12339653757, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.405521] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.405544] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.405562] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.405572] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.405588] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12339653757, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.405603] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.405617] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.405629] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.405638] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.405650] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.405659] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.405668] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037405475) [2025-05-15 09:40:37.410544] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.413909] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.417364] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.420651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.427065] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=12339698286, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.427098] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.427118] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.427133] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.427145] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.427161] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.427160] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12339525793, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.427171] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.427178] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037427052) [2025-05-15 09:40:37.427176] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.427191] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.427208] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.427217] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.427227] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.427236] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.427244] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037427147) [2025-05-15 09:40:37.428346] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.430759] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.437812] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.437833] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6833884, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.437849] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6833884, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.437863] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.437874] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=6833884, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.437889] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.437904] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.437919] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.437933] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.437945] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.437956] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.437971] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037437970, replica_locations:[]}) [2025-05-15 09:40:37.437989] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.438013] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.438026] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:37.438073] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.438119] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11801328597, cache_obj->added_lc()=false, cache_obj->get_object_id()=15799, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.439161] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.439182] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.439198] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.439219] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.439230] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.439244] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.439255] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.439269] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.439283] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.439294] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.439308] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.439322] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037439322, replica_locations:[]}) [2025-05-15 09:40:37.439382] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=6832300, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=277, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.441905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.442576] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=9) [2025-05-15 09:40:37.452020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.462137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.472241] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.478852] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:32, tid:4602}]) [2025-05-15 09:40:37.482354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.492458] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.495353] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [4580][T1002_TimerWK3_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=11] BlockGCTimerTask success(ret=0, cost_time_us=6, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"127.0.0.1:2882", log_dir:"/root/ob/observer/store/clog/tenant_1002", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):195, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-15 09:40:37.502576] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.504509] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.505452] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12339753204, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.505478] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.505507] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.505523] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.505535] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.505551] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12339754261, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.505563] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.505577] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.505586] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.505592] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.505602] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.505612] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.505618] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037505432) [2025-05-15 09:40:37.505939] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.505954] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.505964] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:37.505940] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.505976] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:37.505986] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:37.505998] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:37.506013] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:37.506026] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:37.506138] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [4577][T1002_TimerWK0_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=11] clear dtl interm result cost(us)(clear_cost=10396, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-15 09:40:37.506153] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=1] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:37.507736] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=18][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.507758] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.507771] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=12][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:37.507781] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=10][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:37.507791] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=9][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:37.507809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=10][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:37.507825] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.507836] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C6666-0-0] [lt=10][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5157, ret=-5157) [2025-05-15 09:40:37.507846] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:37.507858] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:37.507866] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:37.507875] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78ca060, start=1747302037507715, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:37.507889] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:37.508013] INFO [STORAGE] runTimerTask (ob_tenant_restore_info_mgr.cpp:281) [4580][T1002_TimerWK3_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=15] refresh restore info [2025-05-15 09:40:37.512686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.517291] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.517311] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.517329] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.517342] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.517353] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.517367] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.517377] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.517386] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.517396] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302037517276, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.517404] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.517414] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.517475] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.517486] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.517507] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.517516] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.522802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.527112] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.527141] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037527106) [2025-05-15 09:40:37.527158] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302037327068, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.527189] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12339798781, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.527205] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.527208] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.527227] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037527204) [2025-05-15 09:40:37.527226] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.527240] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.527240] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302037327145, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.527250] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.527264] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.527263] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12339626288, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.527273] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.527276] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.527284] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037527177) [2025-05-15 09:40:37.527291] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.527304] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.527312] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.527325] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.527334] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.527343] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037527251) [2025-05-15 09:40:37.532911] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.535282] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=13] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:37.539507] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.539530] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6732197, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.539578] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=48][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6732197, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.539592] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.539601] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=6732197, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.539615] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.539631] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.539646] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.539657] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.539669] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.539684] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.539699] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037539699, replica_locations:[]}) [2025-05-15 09:40:37.539716] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.539740] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.539753] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:37.539780] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.539827] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11801430304, cache_obj->added_lc()=false, cache_obj->get_object_id()=15800, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.540898] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.540919] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.540934] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.540949] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.540956] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.540970] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.540984] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.540999] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.541012] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.541024] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.541038] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.541053] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037541052, replica_locations:[]}) [2025-05-15 09:40:37.541111] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6730570, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=278, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.541375] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [4064][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=26] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-05-15 09:40:37.542170] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:37.542331] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=18] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:37.543020] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.553128] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.563233] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.565462] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.565514] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:37.565531] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:37.565554] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:37.565568] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:37.565584] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:37.565944] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.565957] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:37.565969] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:37.569543] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:37.569586] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:37.569962] INFO pn_ratelimit (group.c:653) [3953][TimerWK0_IngressService][T0][Y0-0000000000000000-0-0] [lt=9] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-15 09:40:37.573319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.574182] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6605-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.574204] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6605-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.574225] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6605-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.574240] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6605-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.574252] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.574263] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:37.574275] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.574286] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.574296] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302037574157, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:37.574309] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.574317] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:37.574384] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=10][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:37.574397] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=11][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:37.574409] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=11][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:37.574424] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=14][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:37.574438] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=12][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:37.574450] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=9][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:37.574459] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6605-0-0] [lt=9][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:37.577968] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221553071, support_ssl=false) [2025-05-15 09:40:37.578096] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221553071, sess.client_addr_="172.22.0.1:46494") [2025-05-15 09:40:37.578119] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f620db0, session_id:3221553071, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.578247] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.578289] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.578319] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.578343] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.578360] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.578373] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.578403] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.578423] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553071, proxy_sessid=0) [2025-05-15 09:40:37.578464] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=14][errcode=0] server close connection(sessid=3221553071, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.578477] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553071, proxy_sessid=0) [2025-05-15 09:40:37.578485] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.578516] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9089-0-0] [lt=27] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553071, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.578573] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f620db0, session_id:3221553071, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037577394, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.578597] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553071, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.578610] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a6f620db0, session_id:3221553071, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037577394, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.578629] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553071) [2025-05-15 09:40:37.578654] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553071, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.583426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.593534] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.601441] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [3954][TimerWK1_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.601463] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3954][TimerWK1_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.601478] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3954][TimerWK1_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to get percent [2025-05-15 09:40:37.601496] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3954][TimerWK1_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] failed to calculate global bound size(ret=-5627, ids[i]=1001) [2025-05-15 09:40:37.601515] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [3954][TimerWK1_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.601527] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [3954][TimerWK1_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:37.601615] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [3954][TimerWK1_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=88][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:37.601627] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [3954][TimerWK1_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:37.601635] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3954][TimerWK1_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:37.601644] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3954][TimerWK1_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] failed to get percent [2025-05-15 09:40:37.601652] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3954][TimerWK1_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] failed to calculate global bound size(ret=-5157, ids[i]=1002) [2025-05-15 09:40:37.604556] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.605505] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.605535] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037605498) [2025-05-15 09:40:37.605550] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302037405469, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.605584] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12339853706, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.605604] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.605627] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.605644] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.605656] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.605672] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12339853706, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.605684] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.605697] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.605709] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.605718] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.605730] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.605740] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.605749] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037605562) [2025-05-15 09:40:37.605830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.615941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.617357] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.617378] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.617397] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.617413] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.617424] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.617436] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.617444] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.617454] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.617464] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302037617340, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.617479] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.617497] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.617557] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.617567] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.617577] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.617587] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.620583] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553073, support_ssl=false) [2025-05-15 09:40:37.620642] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553073, sess.client_addr_="172.22.0.1:46500") [2025-05-15 09:40:37.620659] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f620030, session_id:3221553073, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.620685] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553074, support_ssl=false) [2025-05-15 09:40:37.620715] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553074, sess.client_addr_="172.22.0.1:46514") [2025-05-15 09:40:37.620728] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553074, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.620857] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.620884] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.620904] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221553075, support_ssl=false) [2025-05-15 09:40:37.620916] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.620941] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553075, sess.client_addr_="172.22.0.1:46526") [2025-05-15 09:40:37.620954] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=36][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.620955] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553075, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.620965] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.620982] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.620982] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=17][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.621003] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.621011] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.621033] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221553073, proxy_sessid=0) [2025-05-15 09:40:37.621037] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.621057] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.621070] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.621067] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=16][errcode=0] server close connection(sessid=3221553073, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.621081] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553073, proxy_sessid=0) [2025-05-15 09:40:37.621082] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.621089] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.621105] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.621103] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9078-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553073, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.621125] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553074, proxy_sessid=0) [2025-05-15 09:40:37.621163] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=10][errcode=0] server close connection(sessid=3221553074, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.621158] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f620030, session_id:3221553073, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037620785, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.621174] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553074, proxy_sessid=0) [2025-05-15 09:40:37.621179] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553073, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.621183] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:37.621192] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a6f620030, session_id:3221553073, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037620785, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.621210] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553073) [2025-05-15 09:40:37.621196] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BA-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553074, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.621226] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553073, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.621264] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553074, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037620785, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.621283] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553074, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.621305] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553074, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037620785, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.621322] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553074) [2025-05-15 09:40:37.621333] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553074, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.621364] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.621388] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.621407] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=29][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.621428] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.621442] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.621449] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.621470] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.621485] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553075, proxy_sessid=0) [2025-05-15 09:40:37.621521] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=22][errcode=0] server close connection(sessid=3221553075, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.621533] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553075, proxy_sessid=0) [2025-05-15 09:40:37.621540] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.621552] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908A-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553075, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.621598] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553075, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037620785, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.621621] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553075, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.621629] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553075, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037620785, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.621645] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553075) [2025-05-15 09:40:37.621657] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553075, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.622148] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553076, support_ssl=false) [2025-05-15 09:40:37.622244] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=45] sm conn init succ(conn.sessid_=3221553076, sess.client_addr_="172.22.0.1:46538") [2025-05-15 09:40:37.622249] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553077, support_ssl=false) [2025-05-15 09:40:37.622265] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553076, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.622331] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221553077, sess.client_addr_="172.22.0.1:46548") [2025-05-15 09:40:37.622351] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553077, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.622431] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.622470] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.622500] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.622516] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221553078, support_ssl=false) [2025-05-15 09:40:37.622515] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.622518] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.622529] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.622557] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.622545] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.622557] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553078, sess.client_addr_="172.22.0.1:46556") [2025-05-15 09:40:37.622573] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=23] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.622571] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553078, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.622586] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553076, proxy_sessid=0) [2025-05-15 09:40:37.622590] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.622612] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=14][errcode=0] server close connection(sessid=3221553076, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.622625] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553076, proxy_sessid=0) [2025-05-15 09:40:37.622633] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:37.622625] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=36][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.622641] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9079-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553076, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.622650] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=25][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.622680] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=29][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.622700] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.622695] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553076, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037621843, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.622716] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553077, proxy_sessid=0) [2025-05-15 09:40:37.622721] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553076, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.622737] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=7][errcode=0] server close connection(sessid=3221553077, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.622733] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553076, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037621843, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.622750] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553076) [2025-05-15 09:40:37.622751] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553077, proxy_sessid=0) [2025-05-15 09:40:37.622759] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:37.622768] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BB-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553077, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.622766] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553076, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.622803] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553079, support_ssl=false) [2025-05-15 09:40:37.622823] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553077, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037621843, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.622837] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553079, sess.client_addr_="172.22.0.1:46560") [2025-05-15 09:40:37.622850] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553077, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.622850] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553079, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.622865] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553077, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302037621843, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.622878] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.622885] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553077) [2025-05-15 09:40:37.622885] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553080, support_ssl=false) [2025-05-15 09:40:37.622905] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.622903] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553077, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.622927] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.622947] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.622956] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.622970] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.622982] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=6] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623000] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553078, proxy_sessid=0) [2025-05-15 09:40:37.623019] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=8][errcode=0] server close connection(sessid=3221553078, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623018] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553080, sess.client_addr_="172.22.0.1:46564") [2025-05-15 09:40:37.623035] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553078, proxy_sessid=0) [2025-05-15 09:40:37.623043] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:37.623050] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553081, support_ssl=false) [2025-05-15 09:40:37.623040] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de2adb0, session_id:3221553080, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623060] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908B-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553078, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.623092] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553081, sess.client_addr_="172.22.0.1:46576") [2025-05-15 09:40:37.623098] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.623116] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553081, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623138] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.623158] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.623179] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553082, support_ssl=false) [2025-05-15 09:40:37.623201] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553078, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302037621843, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623215] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=56][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.623223] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553078, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.623227] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.623221] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553082, sess.client_addr_="172.22.0.1:46590") [2025-05-15 09:40:37.623237] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.623242] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221553083, support_ssl=false) [2025-05-15 09:40:37.623234] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553078, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302037621843, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.623241] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553082, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623261] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] mark session id unused(conn.sessid_=3221553078) [2025-05-15 09:40:37.623266] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=24] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623275] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553083, sess.client_addr_="172.22.0.1:46600") [2025-05-15 09:40:37.623281] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553079, proxy_sessid=0) [2025-05-15 09:40:37.623286] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553083, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623281] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553078, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.623310] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=13][errcode=0] server close connection(sessid=3221553079, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623361] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=50][errcode=-4018] get session fail(ret=-4018, sessid=3221553079, proxy_sessid=0) [2025-05-15 09:40:37.623371] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:37.623394] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.623398] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.623410] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907A-0-0] [lt=38] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553079, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.623422] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.623438] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.623441] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.623448] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.623451] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.623458] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=10][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.623459] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.623466] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.623467] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.623466] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553079, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623476] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.623485] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623480] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.623486] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553079, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.623506] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.623511] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=24][errcode=-4018] get session fail(ret=-4018, sessid=3221553081, proxy_sessid=0) [2025-05-15 09:40:37.623512] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=28] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623524] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553080, proxy_sessid=0) [2025-05-15 09:40:37.623530] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=8][errcode=0] server close connection(sessid=3221553081, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623519] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.623540] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553081, proxy_sessid=0) [2025-05-15 09:40:37.623541] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=8][errcode=0] server close connection(sessid=3221553080, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623549] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:37.623542] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.623553] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553080, proxy_sessid=0) [2025-05-15 09:40:37.623546] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553079, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.623564] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:37.623567] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553079) [2025-05-15 09:40:37.623559] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BC-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553081, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.623562] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.623572] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907B-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553080, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.623579] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=16][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.623587] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:37.623586] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553079, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.623599] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623607] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553081, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623627] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553081, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.623623] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2adb0, session_id:3221553080, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623630] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=29][errcode=-4018] get session fail(ret=-4018, sessid=3221553083, proxy_sessid=0) [2025-05-15 09:40:37.623634] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553081, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.623648] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553081) [2025-05-15 09:40:37.623644] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:37.623650] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553080, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.623658] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:37.623661] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de2adb0, session_id:3221553080, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.623663] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=18][errcode=0] server close connection(sessid=3221553083, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623670] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:37.623661] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553081, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.623676] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553080) [2025-05-15 09:40:37.623678] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:37.623678] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553083, proxy_sessid=0) [2025-05-15 09:40:37.623686] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:37.623694] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:37.623701] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623713] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553082, proxy_sessid=0) [2025-05-15 09:40:37.623696] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553080, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.623729] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=6][errcode=0] server close connection(sessid=3221553082, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.623714] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908C-0-0] [lt=19] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553083, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.623744] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553082, proxy_sessid=0) [2025-05-15 09:40:37.623751] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:37.623758] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BD-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553082, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:37.623760] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553083, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623777] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221553083, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.623805] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553082, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:37.623812] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553083, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.623829] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553083) [2025-05-15 09:40:37.623841] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553083, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.623830] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553082, proxy_sessid=0, ret=0) [2025-05-15 09:40:37.623864] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=34] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553082, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302037622903, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:37.623883] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553082) [2025-05-15 09:40:37.623895] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553082, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:37.626049] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.627184] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12339899304, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.627210] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.627235] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.627252] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.627266] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.627267] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.627280] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.627290] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.627290] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037627261) [2025-05-15 09:40:37.627297] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037627166) [2025-05-15 09:40:37.627307] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302037527251, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.627334] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12339726811, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.627350] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.627369] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.627386] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.627394] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.627410] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.627417] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.627428] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037627322) [2025-05-15 09:40:37.636157] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.640269] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4289][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=20] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4833632278, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:37.640625] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4289][T1_TimerWK0][T1][Y0-0000000000000000-0-0] [lt=27] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4833632281, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:37.641282] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.641325] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=35][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6630434, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.641350] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6630434, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.641367] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.641386] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=6630434, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.641405] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.641426] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.641448] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.641469] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.641479] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.641510] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.641532] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037641529, replica_locations:[]}) [2025-05-15 09:40:37.641555] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.641587] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.641600] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:37.641648] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.641714] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11801532187, cache_obj->added_lc()=false, cache_obj->get_object_id()=15801, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:37.642668] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=13] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:37.643114] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.643134] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.643152] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.643168] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.643175] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:37.643189] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:37.643206] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:37.643222] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.643237] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.643250] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.643266] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.643277] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037643277, replica_locations:[]}) [2025-05-15 09:40:37.643350] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6628332, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=279, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.646267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.656382] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.657169] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4050][LSUserTblUp6][T0][YB427F000001-000635291D6D159D-0-0] [lt=55][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001, task={tenant_id:1002, ls_id:{id:1}, add_timestamp:1747300526212290}) [2025-05-15 09:40:37.666494] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.676601] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.686709] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.696820] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.704623] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.705581] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12339954235, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.705612] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.705637] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.705657] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.705670] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.705686] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12339954235, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.705701] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.705717] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.705730] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.705738] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.705748] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.705759] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.705766] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037705559) [2025-05-15 09:40:37.706930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.717551] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.717573] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.717593] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.717613] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.717627] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.717641] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.717657] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.717670] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.717680] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302037717526, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.717695] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.717703] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.717772] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.717785] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.717806] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.717815] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.718513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.727228] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.727268] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037727222) [2025-05-15 09:40:37.727287] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302037527171, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.727315] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12339998760, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.727333] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12339826267, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.727333] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.727353] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.727354] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.727365] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.727369] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.727373] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.727384] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.727390] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.727396] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.727400] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.727407] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.727411] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037727303) [2025-05-15 09:40:37.727420] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.727427] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037727319) [2025-05-15 09:40:37.728625] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.735557] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:37.738735] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.741793] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3955][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:37.743479] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.743515] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=33][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6528218, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.743533] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6528218, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.743549] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:37.743568] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.743584] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.743593] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.743609] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.743626] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037743625, replica_locations:[]}) [2025-05-15 09:40:37.743646] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.743672] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.744876] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.744899] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.744916] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.744940] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.744956] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.744966] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.744982] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.744997] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037744997, replica_locations:[]}) [2025-05-15 09:40:37.745058] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6526623, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=280, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.748845] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.755340] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.755358] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:37.755368] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:37.755387] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:37.755398] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-15 09:40:37.755429] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=36] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:37.758960] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.769070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.779183] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.779471] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:53, tid:4291}]) [2025-05-15 09:40:37.789293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.799406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.804684] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.805625] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.805655] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037805618) [2025-05-15 09:40:37.805674] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302037605561, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.805704] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12340053686, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.805723] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.805745] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.805762] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.805770] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.805789] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12340053686, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.805803] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.805819] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.805833] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.805847] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.805857] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.805868] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.805875] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037805689) [2025-05-15 09:40:37.809520] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.817560] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.817580] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.817600] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.817622] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.817637] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.817652] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.817661] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.817676] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302037817536, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.817691] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.817706] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.817774] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.817787] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.817797] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.817814] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.819630] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.820066] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:37.827297] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12340099225, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.827322] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.827347] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.827363] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.827375] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.827387] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.827380] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.827399] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.827407] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037827284) [2025-05-15 09:40:37.827411] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037827375) [2025-05-15 09:40:37.827425] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302037627322, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.827454] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12339926732, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.827477] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.827506] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.827523] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.827531] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.827547] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.827560] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.827566] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037827444) [2025-05-15 09:40:37.829739] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.839850] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.842755] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=7] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:37.845180] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.845205] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6426515, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.845224] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6426515, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.845248] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.845265] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.845274] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.845289] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.845307] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037845306, replica_locations:[]}) [2025-05-15 09:40:37.845326] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.845352] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.846468] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.846500] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.846518] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.846542] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.846557] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.846571] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.846587] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.846602] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037846602, replica_locations:[]}) [2025-05-15 09:40:37.846661] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6425020, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=281, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.849959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.858897] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:37.859421] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4243][T1_Occam][T1][YB427F000001-000635291B4C65D1-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.860069] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.861485] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=8] Garbage Collector is running(seq_=152, gc_interval=10000000) [2025-05-15 09:40:37.861528] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.861642] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=1] gc_check_member_list_ cost time(ret=0, time_us=126) [2025-05-15 09:40:37.861660] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=1) [2025-05-15 09:40:37.861690] INFO [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1776) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=26] gc_candidates push_back success(ret=0, candidate={ls_id_:{id:1}, ls_status_:1, gc_reason_:0}) [2025-05-15 09:40:37.861708] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1798) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=15] ls status is normal, skip(id={id:1}, gc_candidates=[{ls_id_:{id:1}, ls_status_:1, gc_reason_:0}]) [2025-05-15 09:40:37.861721] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=12] execute_gc cost time(ret=0, time_us=14) [2025-05-15 09:40:37.861731] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [4312][T1_GCCollector][T1][Y0-0000000000000000-0-0] [lt=9] ObSafeDestroyHandler start process [2025-05-15 09:40:37.867920] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4461][T1001_TimerWK1_Flush][T1001][Y0-0000000000000000-0-0] [lt=15] ====== traversal_flush timer task ====== [2025-05-15 09:40:37.868095] INFO [MDS] flush (mds_table_mgr.cpp:184) [4461][T1001_TimerWK1_Flush][T1001][YB427F000001-000635291D5C6444-0-0] [lt=17] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, {tablet_id:{id:60157}, rec_scn:{val:1747274416328312026, v:0}}, {tablet_id:{id:403}, rec_scn:{val:1747274416328312027, v:0}}, {tablet_id:{id:101079}, rec_scn:{val:1747274416328312028, v:0}}, {tablet_id:{id:50307}, rec_scn:{val:1747274416328312029, v:0}}, {tablet_id:{id:50272}, rec_scn:{val:1747274416328312030, v:0}}, {tablet_id:{id:383}, rec_scn:{val:1747274416328312031, v:0}}, {tablet_id:{id:238}, rec_scn:{val:1747274416328312032, v:0}}, {tablet_id:{id:60306}, rec_scn:{val:1747274416328312033, v:0}}, {tablet_id:{id:50333}, rec_scn:{val:1747274416328312034, v:0}}, {tablet_id:{id:150}, rec_scn:{val:1747274416328312035, v:0}}, {tablet_id:{id:50238}, rec_scn:{val:1747274416328312036, v:0}}, {tablet_id:{id:406}, rec_scn:{val:1747274416328312037, v:0}}, {tablet_id:{id:50531}, rec_scn:{val:1747274416328312038, v:0}}, {tablet_id:{id:1}, rec_scn:{val:1747274416328312039, v:0}}, {tablet_id:{id:60330}, rec_scn:{val:1747274416328312040, v:0}}, {tablet_id:{id:50319}, rec_scn:{val:1747274416328312041, v:0}}, {tablet_id:{id:50377}, rec_scn:{val:1747274416328312042, v:0}}, {tablet_id:{id:396}, rec_scn:{val:1747274416328312043, v:0}}, {tablet_id:{id:467}, rec_scn:{val:1747274416328312044, v:0}}, {tablet_id:{id:60407}, rec_scn:{val:1747274416328312045, v:0}}, {tablet_id:{id:50430}, rec_scn:{val:1747274416328312046, v:0}}, {tablet_id:{id:50104}, rec_scn:{val:1747274416328312047, v:0}}, {tablet_id:{id:50425}, rec_scn:{val:1747274416328312048, v:0}}, {tablet_id:{id:50480}, rec_scn:{val:1747274416328312049, v:0}}, {tablet_id:{id:60503}, rec_scn:{val:1747274416328312050, v:0}}, {tablet_id:{id:60390}, rec_scn:{val:1747274416328312051, v:0}}, {tablet_id:{id:50436}, rec_scn:{val:1747274416328312052, v:0}}, {tablet_id:{id:50160}, rec_scn:{val:1747274416328312053, v:0}}, {tablet_id:{id:474}, rec_scn:{val:1747274416328312054, v:0}}, {tablet_id:{id:60393}, rec_scn:{val:1747274416328312055, v:0}}, {tablet_id:{id:50451}, rec_scn:{val:1747274416328312056, v:0}}, }, max_consequent_callbacked_scn={val:1747289697900328000, v:0}, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:37.868245] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4461][T1001_TimerWK1_Flush][T1001][YB427F000001-000635291D5C6444-0-0] [lt=145] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-15 09:40:37.870179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.880291] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.890402] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.893334] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4291][T1_TimerWK2_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=0] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:37.900515] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.904743] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:37.905499] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1747302037905481, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-05-15 09:40:37.905700] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12340154193, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.905725] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.905747] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.905763] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.905773] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.905787] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1001}, delta=12340154193, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:37.905800] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.905814] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:37.905825] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:37.905834] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:37.905847] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.905856] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:37.905865] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037905683) [2025-05-15 09:40:37.910624] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.911367] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=26] PNIO [ratelimit] time: 1747302037911364, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-05-15 09:40:37.917627] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.917647] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.917693] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=38][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.917709] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.917723] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.917734] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.917747] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.917758] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302037917605, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.917772] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.917781] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:37.917848] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:37.917858] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:37.917868] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:37.917879] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:37.920731] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.922445] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:37.922450] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=33] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:37.927346] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:37.927375] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302037927341) [2025-05-15 09:40:37.927393] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302037727302, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:37.927417] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12340198729, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:37.927434] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.927447] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12340026236, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:37.927453] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:37.927470] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.927464] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:37.927479] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.927480] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:37.927502] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:37.927506] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.927509] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:37.927513] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:37.927520] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.927520] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037927405) [2025-05-15 09:40:37.927528] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:37.927537] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302037927435) [2025-05-15 09:40:37.930840] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.931171] INFO [CLOG.EXTLOG] resize_log_ext_handler_ (ob_cdc_service.cpp:748) [4308][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=7] finish to resize log external storage handler(current_ts=1747302037931161, tenant_max_cpu=3, valid_ls_v1_count=0, valid_ls_v2_count=0, other_ls_count=0, new_concurrency=0) [2025-05-15 09:40:37.931200] INFO [CLOG.EXTLOG] run1 (ob_cdc_service.cpp:239) [4308][T1_CdcSrv][T1][Y0-0000000000000000-0-0] [lt=28] snapshot_traffic_info once [2025-05-15 09:40:37.935526] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4834773928, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:37.935554] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:37.935564] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1aaefec3a0) [2025-05-15 09:40:37.935577] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4834773928, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:37.935808] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:37.940949] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.946782] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.946804] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6324914, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.946820] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6324914, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.946845] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.946859] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.946878] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.946893] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.946908] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037946907, replica_locations:[]}) [2025-05-15 09:40:37.946925] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:37.946949] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:37.948126] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:37.948146] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.948162] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:37.948180] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.948194] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:37.948202] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:37.948214] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:37.948228] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302037948228, replica_locations:[]}) [2025-05-15 09:40:37.948288] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6323394, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=282, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:37.951060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.961172] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.961896] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=25] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:37.961929] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:37.961943] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:37.961957] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:37.961971] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:37.961978] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=8][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:37.961992] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:37.962003] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:37.962013] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:37.962023] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=9][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:37.962034] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:37.962045] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:37.962058] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:37.962067] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:37.962088] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:37.962099] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.962111] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:37.962127] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=15][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:37.962141] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:37.962152] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:37.962162] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:37.962182] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:37.962204] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:37.962213] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:37.962221] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=8][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:37.962259] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:37.962271] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6240-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:37.962282] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4535][T1002_Occam][T1002][YB427F000001-000635291FAC6240-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:37.962293] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:37.962304] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:37.962315] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:37.962326] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302037961645, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:37.962338] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:37.962348] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, tenant_id=1, columns=0x7f1a84e563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:37.962364] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:37.962452] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1a84e563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:37.962467] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:37.962478] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:37.962496] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:37.962517] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:37.962905] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4531][T1002_Occam][T1002][YB427F000001-000635291EDC65CB-0-0] [lt=23] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.962965] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4531][T1002_Occam][T1002][YB427F000001-000635291EDC65CD-0-0] [lt=18] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.971285] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.979897] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=39][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:35, tid:3961}]) [2025-05-15 09:40:37.981396] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.991509] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:37.995753] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE2-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.995772] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE2-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.995792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE2-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.995808] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE2-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.995839] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE2-0-0] [lt=30][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.995851] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:37.995864] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.995873] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.995881] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302037995726, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:37.995894] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.995903] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:37.995977] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:37.997307] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE3-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.997324] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE3-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:37.997335] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE3-0-0] [lt=7][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:37.997348] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE3-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:37.997360] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE3-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:37.997371] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:37.997382] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:37.997389] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:37.997400] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302037997298, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:37.997412] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:37.997419] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:37.997460] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:37.998741] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:38.001615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.001651] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553090, support_ssl=false) [2025-05-15 09:40:38.001715] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553090, sess.client_addr_="127.0.0.1:45118") [2025-05-15 09:40:38.001731] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553090, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.001894] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.001928] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.001954] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.001987] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=31][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.001999] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.002014] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.002031] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.002049] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553090, proxy_sessid=0) [2025-05-15 09:40:38.002080] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=15][errcode=0] server close connection(sessid=3221553090, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.002096] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553090, proxy_sessid=0) [2025-05-15 09:40:38.002103] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.002118] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907C-0-0] [lt=13] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553090, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=3908101, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.002178] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553090, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038001729, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:138, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.002210] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221553090, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.002220] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553090, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038001729, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:138, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.002237] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553090) [2025-05-15 09:40:38.002256] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553090, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.004808] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.005744] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.005790] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038005736) [2025-05-15 09:40:38.005809] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302037805688, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.005850] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12340253707, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.005873] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.005900] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.005921] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.005935] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.005955] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12340254740, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.005970] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.005986] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.005996] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.006007] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.006019] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.006032] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.006040] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038005825) [2025-05-15 09:40:38.006198] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.006196] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.006218] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.006229] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.006240] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.006249] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:38.006261] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:38.006278] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:38.006292] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:38.006325] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:38.009994] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12] ====== tenant freeze timer task ====== [2025-05-15 09:40:38.011729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.017685] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.017708] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.017727] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.017773] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=44][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.017793] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.017814] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.017826] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.017840] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.017850] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302038017659, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.017867] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.017875] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.017942] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.017956] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.017970] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.017982] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.021844] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.027404] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:38.027444] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get wrs ts(ls_id={id:1}, delta=12340299260, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.027465] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.027497] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.027505] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.027514] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.027527] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.027528] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038027500) [2025-05-15 09:40:38.027544] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.027542] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302037827440, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.027551] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.027562] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12340126767, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.027563] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038027431) [2025-05-15 09:40:38.027579] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.027598] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.027615] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.027624] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.027639] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.027650] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.027657] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038027552) [2025-05-15 09:40:38.031954] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.036093] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4462][T1001_TimerWK2_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=20] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:38.036117] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4462][T1001_TimerWK2_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=18] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=80) [2025-05-15 09:40:38.042065] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.042845] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=6) [2025-05-15 09:40:38.048420] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.048445] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6223279, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.048462] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6223279, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.048479] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.048506] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=6223279, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.048523] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.048541] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.048559] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.048576] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.048589] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.048605] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.048623] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038048622, replica_locations:[]}) [2025-05-15 09:40:38.048643] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.048669] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.048682] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:38.048727] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.048779] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11801939256, cache_obj->added_lc()=false, cache_obj->get_object_id()=15805, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.050021] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.050042] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.050060] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.050076] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.050084] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.050099] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.050115] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.050131] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.050147] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.050166] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.050185] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.050200] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038050199, replica_locations:[]}) [2025-05-15 09:40:38.050267] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=6221414, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=283, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.052176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.062290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.065959] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4578][T1002_TimerWK1_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=23] ====== check clog disk timer task ====== [2025-05-15 09:40:38.065985] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4578][T1002_TimerWK1_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=22] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:38.066426] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.066439] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC96A2-0-0] [lt=28][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302038066408}) [2025-05-15 09:40:38.066459] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:38.066473] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:38.066501] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.066515] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.066864] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.066877] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:38.066894] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.066912] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.069729] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:38.069757] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.069772] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:38.069785] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:38.069794] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:38.069875] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:38.072400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.074992] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:38.075035] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=43] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:38.079355] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553092, support_ssl=false) [2025-05-15 09:40:38.079409] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553092, sess.client_addr_="172.22.0.1:46614") [2025-05-15 09:40:38.079427] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553092, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.079587] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.079619] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.079645] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.079674] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.079687] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.079696] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.079714] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.079729] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553092, proxy_sessid=0) [2025-05-15 09:40:38.079754] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=8][errcode=0] server close connection(sessid=3221553092, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.079765] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553092, proxy_sessid=0) [2025-05-15 09:40:38.079774] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:38.079785] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908D-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553092, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.079829] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553092, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038078956, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.079848] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553092, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.079858] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553092, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038078956, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.079872] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553092) [2025-05-15 09:40:38.079886] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553092, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.080119] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:419, tid:4577}]) [2025-05-15 09:40:38.082513] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.084281] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4463][T1001_TimerWK3_][T1001][Y0-0000000000000000-0-0] [lt=1] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:38.084310] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4463][T1001_TimerWK3_][T1001][Y0-0000000000000000-0-0] [lt=23] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:38.084327] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4463][T1001_TimerWK3_][T1001][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=86) [2025-05-15 09:40:38.092626] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.102736] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.104866] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.105726] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=18] ====== tenant freeze timer task ====== [2025-05-15 09:40:38.105816] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12340354211, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.105841] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.105866] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.105883] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.105897] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.105911] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12340354211, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.105926] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.105942] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.105952] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.105964] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.105974] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.105987] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.105993] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038105796) [2025-05-15 09:40:38.109045] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=15] ====== checkpoint timer task ====== [2025-05-15 09:40:38.109717] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=17] succ to get rec scn(*this={this:0x7f1a94c04030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300521088740, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1007}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:38.109755] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a95378080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a95378080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:38.109932] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=33] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1747274416328312025, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, }, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:38.110014] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=79] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1a979cc1b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1a979d7010, memtable_head:0, memtable_tail:1, t3m:0x7f1aaa0ea030, tables:[0x7f1a952dc080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1a979d8d10, ls_tablet_svr:0x7f1a979cc190}, min_rec_scn={val:1747274408663046000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:38.110050] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=29] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-15 09:40:38.110069] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=16] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:38.110084] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:38.110092] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:38.110111] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=12] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:38.110134] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=20] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:38.110152] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274408663046000, v:0}, checkpoint_scn_in_ls_meta={val:1747274408663046000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:38.110172] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:5167067136}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:5167067136}, lsn={lsn:5167067136}) [2025-05-15 09:40:38.110200] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=27] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:5167067136}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:38.110209] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=8] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-15 09:40:38.112848] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.115522] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:38.115543] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:38.115552] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=8] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:38.115571] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=18] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:38.117769] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.117789] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.117809] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.117827] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.117843] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.117853] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.117869] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.117886] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.117902] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302038117737, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.117917] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.117925] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.118003] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.118017] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.118027] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.118038] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.120296] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=16] gc stale ls task succ [2025-05-15 09:40:38.122100] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553094, support_ssl=false) [2025-05-15 09:40:38.122182] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] sm conn init succ(conn.sessid_=3221553094, sess.client_addr_="172.22.0.1:46626") [2025-05-15 09:40:38.122204] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553094, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.122415] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553095, support_ssl=false) [2025-05-15 09:40:38.122457] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221553095, sess.client_addr_="172.22.0.1:46642") [2025-05-15 09:40:38.122476] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553095, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.122540] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=36] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.122599] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=26] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.122629] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.122652] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.122660] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.122673] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.122691] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.122686] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.122699] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.122717] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=30][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.122726] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.122728] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.122734] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.122749] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553094, proxy_sessid=0) [2025-05-15 09:40:38.122755] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.122771] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553095, proxy_sessid=0) [2025-05-15 09:40:38.122791] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=13][errcode=0] server close connection(sessid=3221553094, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.122792] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=8][errcode=0] server close connection(sessid=3221553095, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.122806] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553094, proxy_sessid=0) [2025-05-15 09:40:38.122806] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553095, proxy_sessid=0) [2025-05-15 09:40:38.122799] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553096, support_ssl=false) [2025-05-15 09:40:38.122814] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.122814] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.122824] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907D-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553095, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.122826] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BE-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553094, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.122834] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.122854] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.122874] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.122879] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553096, sess.client_addr_="172.22.0.1:46650") [2025-05-15 09:40:38.122882] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553095, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038122341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.122893] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553096, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.122912] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553094, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038122341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.122932] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553095, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.122944] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] kill and revert session(conn.sessid_=3221553094, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.122954] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553095, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038122341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.122966] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.122975] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553095) [2025-05-15 09:40:38.122995] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553095, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.123019] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553094, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038122341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.123034] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553094) [2025-05-15 09:40:38.123046] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553094, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.123043] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.123104] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=30] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.123124] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.123144] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.123156] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.123166] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.123189] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.123204] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553096, proxy_sessid=0) [2025-05-15 09:40:38.123233] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=11][errcode=0] server close connection(sessid=3221553096, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.123243] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553096, proxy_sessid=0) [2025-05-15 09:40:38.123252] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.123262] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908E-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553096, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.123321] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553096, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038122341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.123369] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=41] kill and revert session(conn.sessid_=3221553096, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.123384] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553096, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038122341, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.123416] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] mark session id unused(conn.sessid_=3221553096) [2025-05-15 09:40:38.123439] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553096, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.123763] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553097, support_ssl=false) [2025-05-15 09:40:38.123803] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553097, sess.client_addr_="172.22.0.1:46666") [2025-05-15 09:40:38.123816] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553097, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.123865] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221553098, support_ssl=false) [2025-05-15 09:40:38.123945] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221553098, sess.client_addr_="172.22.0.1:46672") [2025-05-15 09:40:38.123946] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.123974] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.123965] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553098, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.123993] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.124014] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.124023] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.124038] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.124050] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124066] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553097, proxy_sessid=0) [2025-05-15 09:40:38.124052] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:38.124086] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=9][errcode=0] server close connection(sessid=3221553097, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124098] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553097, proxy_sessid=0) [2025-05-15 09:40:38.124094] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.124105] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.124117] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.124117] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907E-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553097, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.124145] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.124146] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:38.124164] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553097, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038123400, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124165] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:38.124181] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.124185] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553097, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.124185] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:38.124194] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553097, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038123400, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.124199] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.124215] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.124215] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553097) [2025-05-15 09:40:38.124223] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.124229] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] new mysql sessid created(conn.sessid_=3221553099, support_ssl=false) [2025-05-15 09:40:38.124234] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.124233] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553097, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.124255] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124266] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553098, proxy_sessid=0) [2025-05-15 09:40:38.124277] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553099, sess.client_addr_="172.22.0.1:46688") [2025-05-15 09:40:38.124286] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=8][errcode=0] server close connection(sessid=3221553098, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124298] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553098, proxy_sessid=0) [2025-05-15 09:40:38.124296] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553099, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124316] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=17][errcode=-4016] session is null [2025-05-15 09:40:38.124324] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90BF-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553098, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.124334] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553100, support_ssl=false) [2025-05-15 09:40:38.124372] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553100, sess.client_addr_="172.22.0.1:46694") [2025-05-15 09:40:38.124372] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553098, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038123400, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124386] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553100, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124400] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221553098, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.124409] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553098, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038123400, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.124424] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553098) [2025-05-15 09:40:38.124448] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221553098, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.124499] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.124532] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.124564] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.124581] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221553101, support_ssl=false) [2025-05-15 09:40:38.124584] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.124594] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.124583] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553102, support_ssl=false) [2025-05-15 09:40:38.124602] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.124612] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553101, sess.client_addr_="172.22.0.1:46704") [2025-05-15 09:40:38.124616] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124628] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553099, proxy_sessid=0) [2025-05-15 09:40:38.124629] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553101, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124648] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=9][errcode=0] server close connection(sessid=3221553099, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124660] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553099, proxy_sessid=0) [2025-05-15 09:40:38.124659] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.124667] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.124682] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.124677] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221553102, sess.client_addr_="172.22.0.1:46700") [2025-05-15 09:40:38.124678] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC908F-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553099, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.124696] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553102, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124700] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.124720] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.124727] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.124735] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.124729] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553099, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124752] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553099, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.124752] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124763] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553100, proxy_sessid=0) [2025-05-15 09:40:38.124761] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553099, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.124777] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553099) [2025-05-15 09:40:38.124780] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=8][errcode=0] server close connection(sessid=3221553100, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.124792] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553100, proxy_sessid=0) [2025-05-15 09:40:38.124799] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.124799] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221553099, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.124806] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC907F-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553100, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.124853] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553100, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.124877] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553100, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.124885] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553100, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.124901] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553100) [2025-05-15 09:40:38.124913] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553100, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.124945] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.124941] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.124968] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.124975] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.124989] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.124986] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.125001] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.125002] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.125008] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.125007] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553103, support_ssl=false) [2025-05-15 09:40:38.125010] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.125025] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=16][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.125025] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.125037] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125039] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125047] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553102, proxy_sessid=0) [2025-05-15 09:40:38.125048] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553101, proxy_sessid=0) [2025-05-15 09:40:38.125058] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553103, sess.client_addr_="172.22.0.1:46714") [2025-05-15 09:40:38.125071] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=12][errcode=0] server close connection(sessid=3221553102, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125069] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=9][errcode=0] server close connection(sessid=3221553101, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125083] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553102, proxy_sessid=0) [2025-05-15 09:40:38.125091] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.125084] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553101, proxy_sessid=0) [2025-05-15 09:40:38.125081] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553103, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.125100] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=16][errcode=-4016] session is null [2025-05-15 09:40:38.125097] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9090-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553102, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.125105] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553104, support_ssl=false) [2025-05-15 09:40:38.125111] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C0-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553101, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.125148] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553104, sess.client_addr_="172.22.0.1:46722") [2025-05-15 09:40:38.125144] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553102, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.125157] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553101, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.125171] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553102, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.125166] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de28030, session_id:3221553104, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.125177] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553101, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.125180] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553102, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.125187] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553101, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.125200] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553102) [2025-05-15 09:40:38.125202] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553101) [2025-05-15 09:40:38.125214] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553101, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.125213] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553102, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.125262] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.125288] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.125303] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.125315] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.125329] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.125337] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.125345] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.125350] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125363] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553103, proxy_sessid=0) [2025-05-15 09:40:38.125375] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.125381] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=9][errcode=0] server close connection(sessid=3221553103, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125388] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.125398] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553103, proxy_sessid=0) [2025-05-15 09:40:38.125400] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.125406] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.125408] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.125419] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.125421] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9080-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553103, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.125431] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125444] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553104, proxy_sessid=0) [2025-05-15 09:40:38.125459] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=7][errcode=0] server close connection(sessid=3221553104, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.125470] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553104, proxy_sessid=0) [2025-05-15 09:40:38.125480] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:38.125479] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553103, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.125505] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9091-0-0] [lt=24] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553104, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.125514] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] kill and revert session(conn.sessid_=3221553103, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.125533] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553103, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.125556] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221553103) [2025-05-15 09:40:38.125551] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de28030, session_id:3221553104, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.125569] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553103, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.125588] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35] kill and revert session(conn.sessid_=3221553104, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.125596] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de28030, session_id:3221553104, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038124461, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.125616] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553104) [2025-05-15 09:40:38.125630] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553104, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.127464] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.127506] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=40][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038127457) [2025-05-15 09:40:38.127523] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302037927405, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.127557] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12340398747, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.127565] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.127578] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.127590] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038127561) [2025-05-15 09:40:38.127601] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.127603] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302038027552, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.127619] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.127630] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12340226254, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.127636] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.127648] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.127644] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.127660] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.127661] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.127667] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038127539) [2025-05-15 09:40:38.127678] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.127686] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.127701] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.127712] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.127719] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038127617) [2025-05-15 09:40:38.128507] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:38.128532] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-05-15 09:40:38.128548] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=14] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:38.128567] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:38.128586] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11801018923, flush_io_finished_ret_:0, flush_io_finished_round_:1512, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:38.133076] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.134777] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:38.134800] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.134811] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.136070] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:38.143198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.150443] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.150482] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6121273, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.150519] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=36][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6121273, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.150531] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.150549] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=6121273, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.150568] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.150655] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=82][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.150677] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.150696] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.150711] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.150726] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.150749] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038150746, replica_locations:[]}) [2025-05-15 09:40:38.150772] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.150795] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:283, local_retry_times:283, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:38.150824] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.150838] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:38.150852] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.150865] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.150872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:38.150897] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:38.150913] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.150982] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11802041454, cache_obj->added_lc()=false, cache_obj->get_object_id()=15806, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.152382] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.152411] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.152448] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.152465] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.152482] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.152505] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.152513] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.152532] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.152556] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.152582] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.152611] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.152627] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.152639] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.152655] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038152654, replica_locations:[]}) [2025-05-15 09:40:38.152673] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.152688] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.152702] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.152720] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:38.152729] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:38.152742] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:38.152763] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:38.152780] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:38.152788] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:38.152802] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:38.152814] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:38.152823] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:38.152839] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:38.152852] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:38.152865] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:38.152874] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:38.152886] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:38.152894] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:38.152906] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:38.152926] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:38.152939] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.152952] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:38.152961] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:38.152976] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:38.152989] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=284, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:38.153020] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] will sleep(sleep_us=100000, remain_us=6118662, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=284, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.153319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.163427] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.173541] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.173918] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.174300] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=13] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.174546] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.174763] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.183648] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.191802] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=22] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:38.193417] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:38.193448] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:38.193467] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:38.193480] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:38.193622] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=140][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:38.193629] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:38.193642] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:38.193671] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=27][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:38.193683] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:38.193697] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=13][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:38.193705] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:38.193718] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:38.193726] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:38.193738] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:38.193759] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:38.193757] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.193774] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.193785] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.193798] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:38.193807] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:38.193819] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=9][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:38.193832] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:38.193857] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=19][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:38.193880] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.193892] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.193899] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:38.193920] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:38.193936] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6470-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.193950] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:38.193963] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:38.193977] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:38.193991] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=13][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:38.194003] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302038193159, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:38.194015] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=10][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:38.194028] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:38.194124] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=14][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:38.194140] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:38.194154] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=13][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:38.194163] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=8][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:38.194178] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:38.194193] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=13][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:38.194205] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6470-0-0] [lt=12][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:38.203874] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.204927] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.205861] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.205894] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038205853) [2025-05-15 09:40:38.205913] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302038005824, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.205947] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12340453734, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.205969] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.205995] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.206015] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.206029] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.206046] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12340454793, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.206061] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.206080] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.206092] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.206100] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.206115] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.206128] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.206136] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038205924) [2025-05-15 09:40:38.213988] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.217721] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.217744] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.217761] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.217778] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.217792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.217805] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.217884] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=79][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.217893] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.217906] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302038217702, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.217920] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.217927] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.217981] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.217995] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.218017] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.218025] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.222896] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.222920] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.222939] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.224096] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:38.224105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.224119] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.224131] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.224150] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:38.224163] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:38.224172] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:38.227531] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12340499327, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.227556] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.227644] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.227669] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=114][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.227676] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038227637) [2025-05-15 09:40:38.227682] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.227692] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302038127617, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.227694] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.227711] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.227719] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.227718] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12340326834, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.227732] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038227517) [2025-05-15 09:40:38.227733] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.227753] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.227769] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.227776] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.227787] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.227799] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.227805] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038227707) [2025-05-15 09:40:38.234225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.234835] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:38.234855] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.234865] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.242946] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:38.244340] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.253151] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.253177] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6018547, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.253194] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=6018547, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.253209] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.253218] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=6018547, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.253234] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.253251] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.253269] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.253286] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.253299] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.253318] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.253337] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038253335, replica_locations:[]}) [2025-05-15 09:40:38.253357] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.253378] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:284, local_retry_times:284, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:38.253401] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.253414] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:38.253433] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.253445] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.253452] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:38.253485] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:38.253513] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.253560] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11802144038, cache_obj->added_lc()=false, cache_obj->get_object_id()=15807, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.254459] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.254752] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.254781] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.254814] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.254830] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.254850] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.254861] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.254873] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.254883] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.254897] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.254911] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.254927] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.254936] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.254949] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.254967] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038254966, replica_locations:[]}) [2025-05-15 09:40:38.254986] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.254997] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.255011] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.255023] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:38.255032] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:38.255045] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:38.255065] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:38.255080] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:38.255088] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:38.255101] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:38.255110] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:38.255121] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:38.255137] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:38.255150] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:38.255157] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:38.255169] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:38.255177] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:38.255190] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:38.255202] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:38.255218] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:38.255231] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.255244] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:38.255252] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:38.255267] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:38.255274] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=285, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:38.255301] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] will sleep(sleep_us=100000, remain_us=6016381, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=285, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.255497] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.255520] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=33][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:38.255529] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:38.255549] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:38.255564] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:38.255597] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:38.264975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.267238] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65CF-0-0] [lt=16] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.268815] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=28] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:38.275645] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.285759] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.287018] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=0][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:38.287040] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.287054] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.287075] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.287094] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.287106] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.287123] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.287143] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.287162] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.287179] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.287189] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.287208] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.287233] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.287251] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:38.288950] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:38.288981] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=29][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:38.288992] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=10][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:38.289008] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:38.289024] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:38.289038] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:38.289050] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:38.289062] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:38.289070] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:38.289082] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:38.289090] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:38.289102] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:38.289117] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:38.289124] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:38.289148] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:38.289161] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.289172] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=8][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.289186] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:38.289196] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:38.289211] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:38.289224] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:38.289247] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:38.289269] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.289281] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.289287] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:38.289308] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:38.289324] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.289336] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:38.289348] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:38.289361] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:38.289369] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:38.289381] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302038288708, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:38.289399] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=17][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:38.289406] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:38.289502] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:38.289517] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:38.289528] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:38.289542] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=13][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:38.289550] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=7][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:38.289564] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:38.289575] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C0-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:38.295870] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.304983] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.305940] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12340554238, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.305970] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.305987] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.305997] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.306012] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.306026] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.306041] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12340554238, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.306086] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.306103] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.306117] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.306125] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.306141] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.306153] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.306161] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038305913) [2025-05-15 09:40:38.307566] INFO [SERVER] refresh_cpu_frequency (ob_server.cpp:3776) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=21] Cpu frequency changed(from=2453669, to=2478001) [2025-05-15 09:40:38.307608] INFO [STORAGE] runTimerTask (ob_tenant_memory_printer.cpp:26) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=24] === Run print tenant memory usage task === [2025-05-15 09:40:38.307650] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:96) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14] ====== tenants memory info ====== === TENANTS MEMORY INFO === unmanaged_memory_size= 106078208 [TENANT_MEMORY] tenant_id= 500 mem_tenant_limit= 9223372036854775807 mem_tenant_hold= 580640768 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 508 mem_tenant_limit= 1073741824 mem_tenant_hold= 36556800 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 509 mem_tenant_limit= 1073741824 mem_tenant_hold= 23973888 kv_cache_mem= 0 [TENANT_MEMORY] tenant_id= 1 now= 1747302038307552 active_memstore_used= 119537664 total_memstore_used= 119537664 total_memstore_hold= 119537664 memstore_freeze_trigger_limit= 429496720 memstore_limit= 2147483640 mem_tenant_limit= 5368709120 mem_tenant_hold= 850857984 max_mem_memstore_can_get_now= 2630414336 memstore_alloc_pos= 119537664 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1001 now= 1747302038307552 active_memstore_used= 73400320 total_memstore_used= 73400320 total_memstore_hold= 73400320 memstore_freeze_trigger_limit= 345544580 memstore_limit= 1727722920 mem_tenant_limit= 4319307366 mem_tenant_hold= 699817984 max_mem_memstore_can_get_now= 1965982470 memstore_alloc_pos= 73400320 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [TENANT_MEMORY] tenant_id= 1002 now= 1747302038307552 active_memstore_used= 188743680 total_memstore_used= 188743680 total_memstore_hold= 188743680 memstore_freeze_trigger_limit= 3887376620 memstore_limit= 19436883100 mem_tenant_limit= 38873766298 mem_tenant_hold= 971210752 max_mem_memstore_can_get_now= 36023507354 memstore_alloc_pos= 188743680 memstore_frozen_pos= 0 memstore_reclaimed_pos= 0 [2025-05-15 09:40:38.307762] INFO [STORAGE] print_tenant_usage (ob_tenant_memory_printer.cpp:108) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=19] [CHUNK_MGR] limit= 52856746803 hold= 3215794176 total_hold= 3428843520 used= 3160961024 freelists_hold= 54833152 total_maps= 1125 total_unmaps= 43 large_maps= 131 large_unmaps= 0 huge_maps= 48 huge_unmaps= 38 resident_size= 0 unmanaged_memory_size= 106078208 virtual_memory_used= 4345532416 [OS_PARAMS] vm.max_map_count= 65530 transparent_hugepages= always [CHUNK_MGR] 2 MB_CACHE: hold= 23068672 count= 11 pushes= 2817550 pops= 2817539 maps= 946 unmaps= 5 [CHUNK_MGR] 4 MB_CACHE: hold= 31764480 count= 15 pushes= 1195 pops= 1180 maps= 69 unmaps= 0 [CHUNK_MGR] 6 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 21 unmaps= 0 [CHUNK_MGR] 8 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 3 unmaps= 0 [CHUNK_MGR] 10 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 18 unmaps= 0 [CHUNK_MGR] 12 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 14 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 16 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 4 unmaps= 0 [CHUNK_MGR] 18 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 7 unmaps= 0 [CHUNK_MGR] 20 MB_CACHE: hold= 0 count= 0 pushes= 0 pops= 0 maps= 1 unmaps= 0 [2025-05-15 09:40:38.307791] INFO print (ob_malloc_time_monitor.cpp:37) [3953][TimerWK0_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12] [MALLOC_TIME_MONITOR] show the distribution of ob_malloc's cost_time [MALLOC_TIME_MONITOR] [ 1000, 10000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 10000, 100000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 100000, 1000000): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [MALLOC_TIME_MONITOR] [ 1000000, 9223372036854775807): delta_total_cost_time= 0, delta_count= 0, avg_cost_time= 0 [2025-05-15 09:40:38.307888] WDIAG load_file_to_string (utility.h:673) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:38.307905] WDIAG get_ethernet_speed (utility.cpp:575) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:38.307913] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:38.307926] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:38.316105] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.317792] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.317815] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.317835] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.317853] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.317893] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=39][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.317907] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.317921] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.317930] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.317942] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302038317771, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.317959] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.317967] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.318028] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.318045] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.318060] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.318068] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.322957] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.322980] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.323001] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.324154] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:38.324176] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.324187] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.324205] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:38.324213] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:38.324226] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:38.326235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.327580] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.327620] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038327574) [2025-05-15 09:40:38.327638] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302038127539, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.327668] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12340599813, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.327687] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.327707] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12340427320, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.327710] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.327726] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.327729] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.327750] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.334892] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:38.334914] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.334925] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.336350] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.336354] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:38.341697] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=23] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5319, clean_start_pos=200000, clean_num=200000) [2025-05-15 09:40:38.346465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.354749] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-05-15 09:40:38.354781] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.354795] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:38.354804] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:38.354816] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:38.354901] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=86][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:38.354914] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.354926] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:38.354932] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:38.354942] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:38.354994] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=52][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:38.355422] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.355446] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5916274, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.355464] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5916274, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.355479] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.355498] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=5916274, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.355514] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.355532] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.355547] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.355563] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.355578] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.355595] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.355613] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038355612, replica_locations:[]}) [2025-05-15 09:40:38.355632] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.355653] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:285, local_retry_times:285, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:38.355675] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.355688] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:38.355705] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.355717] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.355724] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:38.355745] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:38.355760] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.355807] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11802246283, cache_obj->added_lc()=false, cache_obj->get_object_id()=15808, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.356592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.356858] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.356889] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.356921] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.356938] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.356954] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.356969] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.356977] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.356992] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.357009] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.357025] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.357042] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.357051] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.357067] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.357083] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038357082, replica_locations:[]}) [2025-05-15 09:40:38.357100] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.357117] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.357126] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.357143] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:38.357151] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:38.357164] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:38.357183] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:38.357199] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:38.357207] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:38.357219] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:38.357226] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:38.357239] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:38.357248] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:38.357263] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:38.357271] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:38.357282] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:38.357294] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:38.357302] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:38.357315] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:38.357329] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:38.357345] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.357361] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:38.357371] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:38.357385] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:38.357392] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=286, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:38.357417] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] will sleep(sleep_us=100000, remain_us=5914264, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=286, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.362627] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-05-15 09:40:38.364068] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=20][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.364113] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=45][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.364126] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=12][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.364140] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.364149] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=8][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:38.364166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=8][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:38.364183] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.364192] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_ReqMemEvict][T1002][YB427F000001-000635291F7C6606-0-0] [lt=8][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5157, ret=-5157) [2025-05-15 09:40:38.364206] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:38.364221] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.364229] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:38.364243] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78ca700, start=1747302038364035, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:38.364258] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:38.364266] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5157) [2025-05-15 09:40:38.364337] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] failed to fetch max rule version(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.364350] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] failed to fetch max rule version(ret=-5157) [2025-05-15 09:40:38.364357] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] failed to sync rule from inner table(ret=-5157) [2025-05-15 09:40:38.364866] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:38.364936] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:38.365004] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:38.365066] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:38.365097] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=15] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:14764, total:14776) [2025-05-15 09:40:38.365134] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:38.365185] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:14764, total:14776) [2025-05-15 09:40:38.365217] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:38.365239] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:38.365256] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:38.365279] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] successfully alloc block(macro_id={[ver=1,mode=0,seq=628773][2nd=1900]}) [2025-05-15 09:40:38.366707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.376824] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.379898] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [3960][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=26] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=3626809794560, fast_delete_log_mode=false) [2025-05-15 09:40:38.386936] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.389155] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=11] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:38.389188] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2025-05-15 09:40:38.389206] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:38.389223] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2025-05-15 09:40:38.389239] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:38.389251] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=12][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:38.389265] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2025-05-15 09:40:38.389276] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:38.389290] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:38.389297] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=7][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:38.389309] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:6094) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=11][errcode=-5019] resolve table failed(ret=-5019) [2025-05-15 09:40:38.389320] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:5649) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=10][errcode=-5019] resolve joined table item failed(ret=-5019) [2025-05-15 09:40:38.389328] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5457) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=7][errcode=-5019] resolve joined table failed(ret=-5019) [2025-05-15 09:40:38.389340] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:38.389347] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=7][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:38.389360] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:38.389367] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=6][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:38.389386] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:38.389398] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.389408] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.389420] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:38.389429] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2025-05-15 09:40:38.389442] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:38.389448] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:38.389465] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:38.389497] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.389511] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.389517] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:38.389554] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:38.389569] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4776][BlackListServic][T1][YB427F000001-000635291F1C630A-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.389585] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4776][BlackListServic][T0][YB427F000001-000635291F1C630A-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2025-05-15 09:40:38.389599] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:38.389608] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:38.389620] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:38.389633] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302038388869, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:38.389648] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:114) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:38.389658] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:38.389674] INFO [STORAGE.TRANS] print_stat_ (ob_black_list.cpp:396) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=13] start to print blacklist info [2025-05-15 09:40:38.389749] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7] ls blacklist refresh finish(cost_time=2314) [2025-05-15 09:40:38.390404] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-15 09:40:38.390408] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [4291][T1_TimerWK2_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=21] get_replay_process success(id={id:1}, submitted_log_size=125824135, unsubmitted_log_size=125824135, replayed_log_size=125824135, unreplayed_log_size=0) [2025-05-15 09:40:38.390427] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=25] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-15 09:40:38.390445] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [4291][T1_TimerWK2_ReplayProcessSt][T1][Y0-0000000000000000-0-0] [lt=19] dump tenant replay process(tenant_id=1, unsubmitted_log_size(MB)=119, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=119, replayed_log_size(MB)=119, last_submitted_log_size(MB)=119, last_replayed_log_size(MB)=119, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-15 09:40:38.390452] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=20] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:38.390467] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=13] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:38.390483] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=14] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:38.392042] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=20] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2025-05-15 09:40:38.392073] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=28][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2025-05-15 09:40:38.392090] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=16][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2025-05-15 09:40:38.392107] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2025-05-15 09:40:38.392123] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:38.392133] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=9][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:38.392149] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2025-05-15 09:40:38.392162] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:38.392169] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:38.392182] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:38.392189] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:38.392202] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:38.392210] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:38.392222] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:38.392242] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:38.392255] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.392270] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=13][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:38.392284] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:38.392293] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:38.392309] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:38.392321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:38.392343] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:38.392365] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.392378] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:38.392384] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:38.392402] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:38.392419] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.392432] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4289][T1_TimerWK0_ReqMemEvict][T1][YB427F000001-000635291C0C63C1-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2025-05-15 09:40:38.392442] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:38.392454] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:38.392462] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:38.392474] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302038391795, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:38.392501] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:38.392509] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:38.392582] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2025-05-15 09:40:38.392594] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] failed to fetch max rule version(ret=-5019) [2025-05-15 09:40:38.392601] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2025-05-15 09:40:38.392953] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=24] Dumping All Cache Objs(alloc_obj_list.count()=1, alloc_obj_list=[{obj_id:15809, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:1, added_to_lc:false, mem_used:25832}]) [2025-05-15 09:40:38.392976] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=22] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:38.395897] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=19] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.395930] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.395936] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=15] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.396522] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.396927] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=25] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.396960] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.396964] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=44] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:38.397051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.397806] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4578][T1002_TimerWK1_PalfGC][T1002][Y0-0000000000000000-0-0] [lt=10] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):13824, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):195, sequence:1}) [2025-05-15 09:40:38.405043] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.405978] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001) [2025-05-15 09:40:38.406002] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.406028] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038405971) [2025-05-15 09:40:38.406046] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302038205923, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.406092] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12340654456, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.406114] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.406141] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.406168] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12340654456, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.406184] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.407167] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.414125] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:38.414146] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.414158] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:38.414184] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:38.417284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.417839] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.417860] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.417875] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.417892] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.417907] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.417921] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.417936] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.417945] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.417960] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302038417823, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.417974] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.417981] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.418038] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.418051] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.418059] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.418068] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.423012] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.423036] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.423048] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.424219] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:38.424241] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.424258] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.424278] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:38.424290] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:38.424304] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:38.427399] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.427644] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12340698912, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.427666] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.427751] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.427781] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038427747) [2025-05-15 09:40:38.427800] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302038227707, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.427824] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12340527474, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.427840] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.434950] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:38.434970] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.434980] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.437516] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.443041] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:38.443387] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.443406] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.443416] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:38.443438] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:38.448644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.457555] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.457582] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5814148, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.457602] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5814148, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.457618] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.457628] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=5814148, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.457646] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.457665] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.457683] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.457700] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.457714] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.457731] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.457747] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038457745, replica_locations:[]}) [2025-05-15 09:40:38.457767] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.457788] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:286, local_retry_times:286, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:38.457809] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.457822] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:38.457840] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.457852] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:38.457858] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:38.457894] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:38.457909] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.457950] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11802348427, cache_obj->added_lc()=false, cache_obj->get_object_id()=15809, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.458761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.459160] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.459191] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.459228] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.459247] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.459272] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.459288] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.459296] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.459311] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.459328] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.459345] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.459360] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.459374] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.459392] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.459407] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038459406, replica_locations:[]}) [2025-05-15 09:40:38.459425] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.459436] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.459451] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:38.459469] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:38.459478] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.459550] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=5812132, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=287, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.468876] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.478997] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.481016] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:32, tid:4630}]) [2025-05-15 09:40:38.489115] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.499229] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.505135] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=42][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.506047] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12340754661, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.506073] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.506092] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.506112] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.506126] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.506145] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12340754661, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.506160] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.506175] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.506189] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.506197] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.506211] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.506223] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.506230] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038506030) [2025-05-15 09:40:38.506323] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.506344] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.506352] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.506365] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.506380] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:38.506389] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:38.506408] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:38.506423] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:38.506457] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:38.509338] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.517962] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.519452] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.523070] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.523092] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.523105] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.524283] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.527691] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.527714] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038527686) [2025-05-15 09:40:38.527727] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302038327653, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.527760] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12340799187, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.527785] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.527807] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.527816] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12340626694, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.527824] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.527837] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.527836] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.527853] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.527853] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.527866] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.527867] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.527874] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.527875] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038527749) [2025-05-15 09:40:38.527884] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.527891] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.527909] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038527804) [2025-05-15 09:40:38.529575] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.535012] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.539686] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.542015] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:38.542250] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:38.542974] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=16] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:38.549795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.559708] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.559737] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5712001, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.559758] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5712001, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.559776] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.559789] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=5712001, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.559806] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.559826] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.559847] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.559866] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.559880] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.559898] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.559903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.559921] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038559918, replica_locations:[]}) [2025-05-15 09:40:38.559944] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.559975] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.559989] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:38.560027] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.560092] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11802450564, cache_obj->added_lc()=false, cache_obj->get_object_id()=15810, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.561557] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.561579] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.561595] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.561610] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.561622] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.561637] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.561652] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.561667] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.561681] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.561693] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.561706] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.561717] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038561715, replica_locations:[]}) [2025-05-15 09:40:38.561789] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=5709893, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=288, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.564413] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4292][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=6] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:38.567328] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.567372] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.567391] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=9108, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:38.567405] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.567418] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:38.567438] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:38.567454] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:38.567470] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.567825] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.567836] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:38.567848] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:38.570009] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.575934] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C6668-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.580120] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.580580] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553114, support_ssl=false) [2025-05-15 09:40:38.580663] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] sm conn init succ(conn.sessid_=3221553114, sess.client_addr_="172.22.0.1:46734") [2025-05-15 09:40:38.580682] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553114, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.580919] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.580953] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.580984] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.581006] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.581020] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.581030] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.581053] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.581072] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553114, proxy_sessid=0) [2025-05-15 09:40:38.581110] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=11][errcode=0] server close connection(sessid=3221553114, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.581122] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553114, proxy_sessid=0) [2025-05-15 09:40:38.581132] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.581145] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C1-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553114, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.581200] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553114, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038580849, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.581227] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553114, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.581242] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553114, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038580849, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.581247] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=32][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:32, tid:4602}]) [2025-05-15 09:40:38.581261] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553114) [2025-05-15 09:40:38.581277] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553114, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.584712] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:38.584751] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=17] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:38.590227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.600336] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.605166] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.606096] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.606134] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038606089) [2025-05-15 09:40:38.606149] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302038406061, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.606187] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12340854870, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.606209] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.606235] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.606254] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.606270] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.606285] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1001}, delta=12340854870, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.606298] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.606312] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.606323] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.606332] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.606345] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.606355] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.606363] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038606163) [2025-05-15 09:40:38.612616] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.617966] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.617987] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.618008] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.618025] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.618036] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.618046] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.618057] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.618070] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.618080] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302038617947, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.618092] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.618101] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.618163] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.618173] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.618184] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.618195] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.620250] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.620276] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:38.620285] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.620298] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:38.620312] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:38.620323] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=9] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1002) [2025-05-15 09:40:38.620338] WDIAG [SQL.PC] run_plan_cache_task (ob_plan_cache.cpp:2766) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] fail to update plan cache memory sys val(ret=-4029) [2025-05-15 09:40:38.620350] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=7] start lib cache evict(tenant_id=1002, mem_hold=2097152, mem_limit=1943688310, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:38.620367] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=14] end lib cache evict(tenant_id=1002, cache_evict_num=0, mem_hold=2097152, mem_limit=1943688310, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:38.620381] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=13] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:38.622733] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.622865] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=12] Dumping All Cache Objs(alloc_obj_list.count()=0, alloc_obj_list=[]) [2025-05-15 09:40:38.622891] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4577][T1002_TimerWK0_PlanCacheEvict][T1002][Y0-0000000000000000-0-0] [lt=24] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:38.623136] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.623158] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.623172] WDIAG [CLOG] nonblock_get_leader (ob_location_adapter.cpp:127) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] location adapter nonblock_get_leader failed(ret=-4721, id=1, leader="0.0.0.0:0") [2025-05-15 09:40:38.623185] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.623737] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553116, support_ssl=false) [2025-05-15 09:40:38.623788] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553116, sess.client_addr_="172.22.0.1:46740") [2025-05-15 09:40:38.623805] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553116, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.624060] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.624094] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.624120] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.624149] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.624169] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.624179] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.624204] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.624218] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553116, proxy_sessid=0) [2025-05-15 09:40:38.624246] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=9][errcode=0] server close connection(sessid=3221553116, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.624261] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553116, proxy_sessid=0) [2025-05-15 09:40:38.624270] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:38.624266] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553117, support_ssl=false) [2025-05-15 09:40:38.624279] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9081-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553116, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.624319] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221553117, sess.client_addr_="172.22.0.1:46748") [2025-05-15 09:40:38.624328] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553116, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038623990, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.624348] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553116, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.624340] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553117, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.624356] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553116, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038623990, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.624376] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553116) [2025-05-15 09:40:38.624390] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553116, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.624480] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.624531] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=28] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.624513] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553118, support_ssl=false) [2025-05-15 09:40:38.624568] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.624585] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35] sm conn init succ(conn.sessid_=3221553118, sess.client_addr_="172.22.0.1:46764") [2025-05-15 09:40:38.624591] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=22][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.624606] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.624615] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.624611] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553118, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.624644] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.624664] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553117, proxy_sessid=0) [2025-05-15 09:40:38.624707] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=14][errcode=0] server close connection(sessid=3221553117, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.624721] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553117, proxy_sessid=0) [2025-05-15 09:40:38.624729] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.624745] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9092-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553117, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.624780] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.624806] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553117, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038623990, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.624830] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.624831] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553117, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.624844] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553117, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302038623990, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.624860] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553117) [2025-05-15 09:40:38.624869] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.624873] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553117, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.624901] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=29][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.624921] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.624932] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.624965] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=16] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.624988] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221553118, proxy_sessid=0) [2025-05-15 09:40:38.625130] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=99][errcode=0] server close connection(sessid=3221553118, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.625152] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221553118, proxy_sessid=0) [2025-05-15 09:40:38.625168] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:38.625187] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C2-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553118, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.625232] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553119, support_ssl=false) [2025-05-15 09:40:38.625262] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553118, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038623990, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.625285] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221553119, sess.client_addr_="172.22.0.1:46776") [2025-05-15 09:40:38.625289] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553118, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.625300] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553119, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.625300] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553118, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038623990, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.625320] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553118) [2025-05-15 09:40:38.625337] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553118, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.625380] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553120, support_ssl=false) [2025-05-15 09:40:38.625417] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553120, sess.client_addr_="172.22.0.1:46788") [2025-05-15 09:40:38.625421] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.625434] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553120, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.625450] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.625463] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.625480] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.625497] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.625505] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.625518] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.625533] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553119, proxy_sessid=0) [2025-05-15 09:40:38.625551] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=7][errcode=0] server close connection(sessid=3221553119, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.625565] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553119, proxy_sessid=0) [2025-05-15 09:40:38.625572] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.625571] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.625586] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9082-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553119, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.625605] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.625624] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.625644] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.625651] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.625665] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.625652] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553121, support_ssl=false) [2025-05-15 09:40:38.625677] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.625690] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553120, proxy_sessid=0) [2025-05-15 09:40:38.625685] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553119, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038625050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.625708] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553119, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.625709] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=9][errcode=0] server close connection(sessid=3221553120, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.625722] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553120, proxy_sessid=0) [2025-05-15 09:40:38.625730] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.625722] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553119, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038625050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.625738] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553119) [2025-05-15 09:40:38.625739] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221553121, sess.client_addr_="172.22.0.1:46804") [2025-05-15 09:40:38.625737] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9093-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553120, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.625757] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221553119, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.625759] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553121, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.625784] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553120, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038625050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.625810] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553120, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.625825] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553120, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038625050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.625844] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553120) [2025-05-15 09:40:38.625857] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553120, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.625906] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553122, support_ssl=false) [2025-05-15 09:40:38.625913] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.625944] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221553122, sess.client_addr_="172.22.0.1:46814") [2025-05-15 09:40:38.625957] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.625960] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553122, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.625996] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=40][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.626026] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.626029] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553123, support_ssl=false) [2025-05-15 09:40:38.626037] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.626049] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.626068] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553123, sess.client_addr_="172.22.0.1:46818") [2025-05-15 09:40:38.626071] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626089] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553121, proxy_sessid=0) [2025-05-15 09:40:38.626085] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553123, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:124}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.626110] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553124, support_ssl=false) [2025-05-15 09:40:38.626118] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=12][errcode=0] server close connection(sessid=3221553121, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626129] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553121, proxy_sessid=0) [2025-05-15 09:40:38.626137] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.626144] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553124, sess.client_addr_="172.22.0.1:46822") [2025-05-15 09:40:38.626149] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C3-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553121, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.626166] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553124, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.626187] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.626202] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553121, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038625050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.626224] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.626228] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553121, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.626236] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.626241] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553121, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302038625050, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.626255] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.626261] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553121) [2025-05-15 09:40:38.626263] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.626274] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.626293] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626288] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553121, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.626310] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553122, proxy_sessid=0) [2025-05-15 09:40:38.626328] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=8][errcode=0] server close connection(sessid=3221553122, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626341] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553122, proxy_sessid=0) [2025-05-15 09:40:38.626339] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.626349] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:38.626357] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9083-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553122, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.626368] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.626386] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.626384] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.626401] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.626401] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553122, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.626409] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.626409] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.626423] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.626424] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553122, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.626424] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.626436] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626440] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.626439] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553122, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.626449] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.626449] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553123, proxy_sessid=0) [2025-05-15 09:40:38.626457] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553122) [2025-05-15 09:40:38.626460] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.626472] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=11][errcode=0] server close connection(sessid=3221553123, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626479] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626483] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553123, proxy_sessid=0) [2025-05-15 09:40:38.626479] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221553122, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.626507] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=28][errcode=-4018] get session fail(ret=-4018, sessid=3221553124, proxy_sessid=0) [2025-05-15 09:40:38.626508] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=26][errcode=-4016] session is null [2025-05-15 09:40:38.626516] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9094-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553123, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.626525] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=8][errcode=0] server close connection(sessid=3221553124, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.626535] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553124, proxy_sessid=0) [2025-05-15 09:40:38.626542] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.626549] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C4-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553124, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.626569] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553123, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.626604] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553124, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.626628] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553123, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.626638] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221553124, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.626650] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553123, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:124}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.626670] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553123) [2025-05-15 09:40:38.626691] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553123, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.626721] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553124, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.626734] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553124) [2025-05-15 09:40:38.626751] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221553124, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.626842] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221553125, support_ssl=false) [2025-05-15 09:40:38.626879] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553125, sess.client_addr_="172.22.0.1:46826") [2025-05-15 09:40:38.626885] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553126, support_ssl=false) [2025-05-15 09:40:38.626896] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553125, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.626931] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221553126, sess.client_addr_="172.22.0.1:46832") [2025-05-15 09:40:38.626946] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553126, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.627065] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.627094] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.627113] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.627139] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=26][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.627152] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.627166] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.627180] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.627196] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553125, proxy_sessid=0) [2025-05-15 09:40:38.627193] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:38.627221] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:38.627216] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=9][errcode=0] server close connection(sessid=3221553125, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.627233] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553125, proxy_sessid=0) [2025-05-15 09:40:38.627235] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:38.627241] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.627249] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:38.627263] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:38.627253] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9084-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553125, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.627270] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:38.627290] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.627300] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553126, proxy_sessid=0) [2025-05-15 09:40:38.627302] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553125, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.627322] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=12][errcode=0] server close connection(sessid=3221553126, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.627324] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553125, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.627335] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553126, proxy_sessid=0) [2025-05-15 09:40:38.627342] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:38.627337] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553125, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302038626106, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.627356] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553125) [2025-05-15 09:40:38.627350] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9095-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553126, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:38.627379] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] connection close(sessid=3221553125, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.627397] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553126, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302038627162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:38.627422] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553126, proxy_sessid=0, ret=0) [2025-05-15 09:40:38.627431] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553126, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302038627162, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:38.627446] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553126) [2025-05-15 09:40:38.627464] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221553126, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:38.627755] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=12340899327, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.627783] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.627809] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.627830] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.627846] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.627864] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.627867] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:38.627878] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.627885] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038627732) [2025-05-15 09:40:38.627886] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1001, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302038627865, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302038627865, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:38.627926] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.627959] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038627922) [2025-05-15 09:40:38.627976] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302038427813, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.628006] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] get wrs ts(ls_id={id:1}, delta=12340726834, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.628024] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.628046] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.628063] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.628070] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.628082] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.628095] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.628103] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038627993) [2025-05-15 09:40:38.632846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.642967] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.643132] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:38.653080] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.658411] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EAF5C-0-0] [lt=29][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302038287230}) [2025-05-15 09:40:38.661951] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.661985] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5609763, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.662008] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5609763, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.662026] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.662038] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=5609763, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.662057] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.662080] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:38.662096] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.662117] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.662127] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.662147] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.662172] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038662169, replica_locations:[]}) [2025-05-15 09:40:38.662195] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.662226] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.662242] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:38.662286] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.662349] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11802552825, cache_obj->added_lc()=false, cache_obj->get_object_id()=15811, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:38.663192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.663829] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.663855] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.663875] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.663886] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:38.663895] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:38.663912] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:38.663930] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.663947] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.663956] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.663975] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.663988] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038663987, replica_locations:[]}) [2025-05-15 09:40:38.664069] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=5607613, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=289, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.673303] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.683416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.686228] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=79] tx gc loop thread is running(MTL_ID()=1001) [2025-05-15 09:40:38.686267] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=37] try gc retain ctx [2025-05-15 09:40:38.686277] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-05-15 09:40:38.686307] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=7] finish update min start scn(max_decided_scn={val:1747289697900328000, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289695031638001, v:0}, keep_alive_scn_:{val:1747289695031638001, v:0}, update_ts_:1747302038686216}) [2025-05-15 09:40:38.686906] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4505][T1001_TxLoopWor][T1001][Y0-0000000000000000-0-0] [lt=26] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1001, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:38.693531] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.703644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.705238] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.705904] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4491][T1001_LCLSender][T1001][Y0-0000000000000000-0-0] [lt=32] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1aa57f82b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:38.706178] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12340954071, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.706210] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.706238] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.706261] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.706272] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.706292] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12340954071, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.706309] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.706326] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.706337] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.706350] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.706361] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.706374] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.706382] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038706149) [2025-05-15 09:40:38.713756] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.718124] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.718149] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.718173] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.718198] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.718214] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.718232] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.718254] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.718264] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302038718093, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.718283] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.718291] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.718373] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.718388] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.718397] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.718411] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.722989] INFO [CLOG] do_thread_task_ (ob_remote_fetch_log_worker.cpp:232) [4309][T1_RFLWorker][T1][YB427F000001-000635291B0C6112-0-0] [lt=24] ObRemoteFetchWorker is running(thread_index=0) [2025-05-15 09:40:38.723211] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.723236] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.723256] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.723283] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", [SlidingWindow]={palf_id:1, self:"127.0.0.1:2882", lsn_allocator:{max_log_id:6576040, max_lsn:2944224980, max_scn:{val:1747289697927760000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:2944224391}, reuse_lsn:{lsn:2944224980}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:2944224980}}, last_submit_lsn:{lsn:2944224391}, last_submit_end_lsn:{lsn:2944224980}, last_submit_log_id:6576040, last_submit_log_pid:4, max_flushed_lsn:{lsn:2944224391}, max_flushed_end_lsn:{lsn:2944224980}, max_flushed_log_pid:4, committed_end_lsn:{lsn:2944224391}, last_slide_log_id:6576039, last_slide_scn:{val:1747289697727835001, v:0}, last_slide_lsn:{lsn:2944224269}, last_slide_end_lsn:{lsn:2944224391}, last_slide_log_pid:4, last_slide_log_accum_checksum:2881944827, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f1ab0f48530}, [StateMgr]={this:0x7f1ab0f4c9b0, self:"127.0.0.1:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:-1, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7f1ab0f53470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"127.0.0.1:2882", alive_paxos_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:2442153277}, prev_mode_pid:4}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:4, config_seq:5}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f1ab0f49c70}, [ModeMgr]={palf_id:1, self:"127.0.0.1:2882", applied_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:2818400256}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, log_config_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:2442153277}, prev_mode_pid:4}, log_snapshot_meta:{version:2, base_lsn:{lsn:2818400256}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:86016}, readable_log_tail:{lsn:86016}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:2952609792}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/meta", dir_fd:125, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:67018752, block_header_serialize_buf_:0x7f1ab0f4ebb8, flashback_version:0}, log_storage:{log_tail:{lsn:2944224980}, readable_log_tail:{lsn:2944224980}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:43, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:0, max_block_id:44, curr_writable_block_id:43}, logical_block_size_:67104768, curr_block_writable_size_:8384812, block_header_serialize_buf_:0x7f1ab0f508b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:-1, this:0x7f1ab0f4d0f0}, [Reconfirm]={palf_id:1, self:"127.0.0.1:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f1ab0f4cb30}) [2025-05-15 09:40:38.723536] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=249][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:38.723550] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", ack_info_list=[]) [2025-05-15 09:40:38.723572] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:38.723589] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16] current log_task status(palf_id=1, self="127.0.0.1:2882", log_id=6576040, log_task={header:{begin_lsn:{lsn:2944224391}, end_lsn:{lsn:2944224980}, log_id:6576040, min_scn:{val:1747289697927760000, v:0}, max_scn:{val:1747289697927760000, v:0}, data_len:533, proposal_id:4, prev_lsn:{lsn:2944224269}, prev_proposal_id:4, committed_end_lsn:{lsn:2944224391}, data_checksum:1008861627, accum_checksum:3689732321, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1747300517815094, freeze_ts:1747300517815094, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:-1747300517815095, submit_to_flush cost time:0}, ack_info_list=[]) [2025-05-15 09:40:38.723627] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:396) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=29][errcode=-4018] fail to get server region(ret=-4018, server="127.0.0.1:2882") [2025-05-15 09:40:38.723643] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4018] get_server_region failed(server="127.0.0.1:2882") [2025-05-15 09:40:38.723652] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:398) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:38.723668] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2766) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:38.725373] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.727797] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.727826] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038727791) [2025-05-15 09:40:38.727842] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302038527745, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.727924] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=69] get wrs ts(ls_id={id:1}, delta=12340999647, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.727940] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12340827154, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.727943] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.727963] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.727964] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.727982] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.727984] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.727991] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.727999] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.728007] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.728016] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.728018] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.728025] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038727911) [2025-05-15 09:40:38.728027] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.728039] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.728051] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038727925) [2025-05-15 09:40:38.735497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.741881] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3953][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=22] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:38.742277] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:38.745610] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.755656] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.755686] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:38.755702] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:38.755726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.755808] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=106][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:38.755828] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:38.755855] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:38.764195] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.764220] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5507503, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.764269] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=50][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5507503, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.764296] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.764315] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.764324] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.764342] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.764362] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038764361, replica_locations:[]}) [2025-05-15 09:40:38.764383] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.764410] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.765617] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.765649] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.765680] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=30][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.765708] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.765725] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.765735] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.765754] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.765769] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038765769, replica_locations:[]}) [2025-05-15 09:40:38.765849] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=5505833, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=290, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.765849] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.775964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.786077] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.796192] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.805318] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=26] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:38.805342] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:38.805352] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.806212] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.806243] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038806205) [2025-05-15 09:40:38.806264] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302038606162, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.806295] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12341054580, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.806304] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.806315] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.806339] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.806358] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.806369] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.806388] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12341054580, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.806403] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.806421] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.806431] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.806444] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.806455] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.806468] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.806475] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038806281) [2025-05-15 09:40:38.808370] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4486][T1001_LockWaitM][T1001][Y0-0000000000000000-0-0] [lt=21] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:38.816414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.818142] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.818165] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.818186] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.818208] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.818224] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.818240] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.818249] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.818267] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302038818116, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.818285] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.818294] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.818368] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.818383] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.818391] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.818404] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.820126] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:38.823246] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=8] LogLoopThread round_cost_time(us)(round_cost_time=503) [2025-05-15 09:40:38.823270] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.823281] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.823292] WDIAG [PALF] get_leader_from_cache_ (log_sliding_window.cpp:2149) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock_get_leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:38.823303] WDIAG [PALF] try_fetch_log (log_sliding_window.cpp:1973) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_fetch_log_dst failed or invalid(ret=-4721, palf_id=1, self="127.0.0.1:2882", fetch_log_dst="0.0.0.0:0") [2025-05-15 09:40:38.823313] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.826529] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.827436] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4463][T1001_TimerWK3_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=21] ====== check clog disk timer task ====== [2025-05-15 09:40:38.827456] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4463][T1001_TimerWK3_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=17] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:38.827865] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12341099104, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.827887] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.827912] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.827931] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.827940] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.827959] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.827974] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.827981] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038827852) [2025-05-15 09:40:38.827988] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.828013] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038827983) [2025-05-15 09:40:38.828030] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302038627993, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.828058] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12340927671, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.828077] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.828097] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.828114] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.828122] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.828139] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.828152] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.828159] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038828049) [2025-05-15 09:40:38.836649] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.843225] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=22] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:38.846762] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.856871] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.859464] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=23] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:38.865975] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.865998] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5405722, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.866011] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5405722, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.866037] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.866055] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.866065] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.866089] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.866109] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038866108, replica_locations:[]}) [2025-05-15 09:40:38.866130] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.866157] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.866977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.867287] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.867310] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.867329] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.867353] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.867370] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.867379] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.867397] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.867409] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038867408, replica_locations:[]}) [2025-05-15 09:40:38.867470] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=5404211, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=291, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.877090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.887203] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.897319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.899659] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=18] ====== tenant freeze timer task ====== [2025-05-15 09:40:38.905353] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:38.906279] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=12341154032, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.906307] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.906331] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.906351] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.906360] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.906375] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12341155090, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:38.906391] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.906408] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:38.906456] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=49][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:38.906464] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:38.906480] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.906503] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:38.906510] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038906265) [2025-05-15 09:40:38.907438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.913101] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1747302038913098, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-05-15 09:40:38.914268] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.917560] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.918123] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.918144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.918162] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.918184] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.918200] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.918215] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.918225] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.918241] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302038918104, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.918257] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.918266] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:38.918328] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:38.918343] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:38.918351] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:38.918366] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:38.918984] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=25] PNIO [ratelimit] time: 1747302038918981, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007617, add_bytes: 0 [2025-05-15 09:40:38.923328] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:38.923350] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:38.923363] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:38.924898] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:38.924897] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:38.927673] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.927917] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:38.927952] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302038927911) [2025-05-15 09:40:38.927973] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302038727911, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:38.927999] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12341199605, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:38.928019] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.928042] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:38.928054] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.928050] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12341027112, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:38.928068] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.928069] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:38.928081] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.928091] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:38.928087] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:38.928104] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:38.928105] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038927989) [2025-05-15 09:40:38.928112] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:38.928131] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.928144] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:38.928152] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302038928039) [2025-05-15 09:40:38.935917] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4837766394, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:38.936011] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=27] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:38.936029] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=18] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4837766394, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:38.936705] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.936730] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.936776] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=45][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.936794] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.936819] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.936837] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.936851] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.936872] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.936887] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.936912] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:38.936928] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:38.936936] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=7][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:38.936965] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.936984] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.937000] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.937023] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.937039] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.937055] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=17] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.937065] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.937081] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038937080, replica_locations:[]}) [2025-05-15 09:40:38.937103] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:38.937159] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:38.937177] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.937215] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=38] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:38.937239] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=24][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:38.937265] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=26] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=126, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:38.937292] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=27][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:38.937319] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=24][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:38.937327] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:38.937785] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.942532] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:38.947902] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.948109] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:38.958013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.959191] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:38.959220] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=24] temporary file module use no blocks [2025-05-15 09:40:38.959231] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:38.959251] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:38.959271] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11801849614, flush_io_finished_ret_:0, flush_io_finished_round_:1513, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:38.960829] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=15] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:38.967603] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=34][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.967627] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5304092, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.967641] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5304092, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.967707] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.967724] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.967737] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.967758] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.967770] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038967769, replica_locations:[]}) [2025-05-15 09:40:38.967790] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:38.967817] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:38.968126] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.969097] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:38.969122] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.969141] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:38.969165] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.969182] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:38.969191] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:38.969209] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:38.969220] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302038969220, replica_locations:[]}) [2025-05-15 09:40:38.969293] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=5302388, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=292, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:38.969762] INFO [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:238) [4650][T1002_ArcSender][T1002][YB427F000001-000635291E9C6112-0-0] [lt=21] ObArchiveSender is running(thread_index=4) [2025-05-15 09:40:38.978244] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.982040] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:45, tid:3961}]) [2025-05-15 09:40:38.988356] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.995808] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE4-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.995829] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE4-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.995846] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE4-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.995867] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE4-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.995878] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE4-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.995894] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:38.995905] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.995945] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.995955] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302038995781, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:38.995971] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.995980] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:38.996059] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:38.997328] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE5-0-0] [lt=15][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.997348] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE5-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:38.997360] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE5-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:38.997376] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE5-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:38.997384] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE5-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:38.997398] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:38.997408] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:38.997421] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:38.997428] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302038997317, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:38.997442] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:38.997449] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:38.997496] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:38.998469] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:38.998802] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:39.005411] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.006325] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.006355] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039006321) [2025-05-15 09:40:39.006376] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302038806281, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.006402] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341254494, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.006411] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.006422] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.006443] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.006462] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.006471] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.006485] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12341254494, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.006502] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.006508] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.006520] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.006523] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.006536] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.006537] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:39.006544] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.006547] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:39.006559] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:39.006557] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.006568] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.006569] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:39.006575] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039006389) [2025-05-15 09:40:39.006593] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:39.006611] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:39.006660] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:39.008590] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.018227] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.018250] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.018268] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.018289] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.018306] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.018316] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.018333] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.018342] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.018358] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302039018204, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.018368] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.018382] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.018445] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.018460] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.018469] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.018485] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.018701] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.023385] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.023409] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.023421] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.027975] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:39.028010] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24] get wrs ts(ls_id={id:1}, delta=12341300076, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.028032] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.028054] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.028066] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.028081] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.028094] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.028107] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.028114] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039027999) [2025-05-15 09:40:39.028108] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.028139] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039028103) [2025-05-15 09:40:39.028159] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302038828045, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.028184] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12341127583, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.028202] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.028224] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.028240] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.028249] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.028266] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.028279] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.028286] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039028174) [2025-05-15 09:40:39.028813] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.038923] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.043321] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-05-15 09:40:39.049043] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.059161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.068270] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:39.068280] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC96A9-0-0] [lt=30][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302039068255}) [2025-05-15 09:40:39.068303] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:39.068318] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:39.068337] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:39.068702] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.068717] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:39.068731] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:39.068747] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14] gts statistics(tenant_id=1002, gts_rpc_cnt=0, get_gts_cache_cnt=9142, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:39.069272] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.069421] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.069446] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5202279, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.069464] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5202279, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.069482] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.069509] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=5202279, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.069528] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.069549] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.069568] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.069586] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.069596] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.069615] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.069628] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039069627, replica_locations:[]}) [2025-05-15 09:40:39.069648] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.069675] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.069690] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.069714] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.069765] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11802960242, cache_obj->added_lc()=false, cache_obj->get_object_id()=15815, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.069956] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:39.069987] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.070003] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:39.070013] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:39.070026] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:39.070108] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=81][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:39.070982] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.071007] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.071024] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.071042] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.071050] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.071066] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.071084] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.071096] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.071112] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.071121] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.071138] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.071149] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039071148, replica_locations:[]}) [2025-05-15 09:40:39.071216] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=5200465, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=293, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.079386] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.081962] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553133, support_ssl=false) [2025-05-15 09:40:39.082047] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] sm conn init succ(conn.sessid_=3221553133, sess.client_addr_="172.22.0.1:39666") [2025-05-15 09:40:39.082068] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553133, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.082249] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.082288] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.082319] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.082344] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.082361] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.082369] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.082395] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.082417] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553133, proxy_sessid=0) [2025-05-15 09:40:39.082461] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=16][errcode=0] server close connection(sessid=3221553133, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.082476] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553133, proxy_sessid=0) [2025-05-15 09:40:39.082484] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.082508] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C5-0-0] [lt=19] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553133, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.082569] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553133, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302039081907, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.082596] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553133, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.082613] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553133, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302039081907, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.082634] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553133) [2025-05-15 09:40:39.082652] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553133, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.089449] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:39.089505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.089504] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=44] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:39.099720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=121][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.105471] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.105713] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2025-05-15 09:40:39.105763] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=48][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2025-05-15 09:40:39.105818] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=54][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2025-05-15 09:40:39.105838] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=19][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2025-05-15 09:40:39.105856] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=15][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:39.105871] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:39.105886] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2025-05-15 09:40:39.105901] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=13][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:39.105910] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=8][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:39.105928] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=18][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:39.105940] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:39.105955] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:39.105964] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:39.105971] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=7][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:39.105997] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=14][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:39.106011] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.106024] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.106038] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=12][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:39.106047] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1, ret=-5019) [2025-05-15 09:40:39.106066] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:39.106074] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:39.106097] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:39.106120] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:39.106134] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=14][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:39.106141] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:39.106162] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:39.106179] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.106193] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4298][T1_DupTbLease][T1][YB427F000001-000635291F8C6190-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1"}, aret=-5019, ret=-5019) [2025-05-15 09:40:39.106212] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:39.106221] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:39.106235] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:39.106243] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302039105444, sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:39.106259] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:39.106269] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1) [2025-05-15 09:40:39.106357] WDIAG [SHARE] get_duplicate_ls_status_info (ob_ls_status_operator.cpp:663) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select * from __all_ls_status where tenant_id = 1 and flag like "%DUPLICATE%" order by ls_id limit 1, tenant_id=1, exec_tenant_id=1, need_member_list=false) [2025-05-15 09:40:39.106376] WDIAG [STORAGE.DUP_TABLE] refresh_dup_ls_ (ob_dup_table_util.cpp:110) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] get duplicate ls status info failed(ret=-5019, tmp_dup_ls_status_info={tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}) [2025-05-15 09:40:39.106399] WDIAG [STORAGE.DUP_TABLE] execute_for_dup_ls_ (ob_dup_table_util.cpp:202) [4298][T1_DupTbLease][T1][Y0-0000000000000000-0-0] [lt=23][errcode=0] refresh dup ls failed(tmp_ret=-5019, this={tenant_id_:1, dup_table_scan_timer_:0x7f1ac243dae8, dup_loop_worker_:0x7f1ac27066f0, min_dup_ls_status_info_:{tenant_id:0, ls_id:{id:-1}, ls_group_id:18446744073709551615, status:"UNKNOWN", unit_group_id:18446744073709551615, primary_zone:"", flag:{flag:0, is_duplicate:false, is_block_tablet_in:false}}, tenant_schema_dup_tablet_set_.size():0, scan_task_execute_interval_:10000000, last_dup_ls_refresh_time_:0, last_dup_schema_refresh_time_:0, last_scan_task_succ_time_:1747302027100973, max_execute_interval_:12003548}) [2025-05-15 09:40:39.106410] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341355050, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.106433] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.106457] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.106472] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.106494] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.106514] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1001}, delta=12341355050, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.106531] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.106550] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.106565] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.106573] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.106589] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.106604] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.106611] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039106391) [2025-05-15 09:40:39.109830] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.115653] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:39.115677] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:39.115686] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=8] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:39.115747] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=61] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:39.118170] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.118190] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.118205] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.118222] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.118234] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.118249] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.118258] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.118271] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.118281] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302039118155, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.118299] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.118306] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.118355] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.118370] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.118378] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.118388] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.119939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.120356] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:39.123448] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.123472] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.123486] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.125457] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553136, support_ssl=false) [2025-05-15 09:40:39.125595] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221553136, sess.client_addr_="172.22.0.1:39678") [2025-05-15 09:40:39.125619] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221553136, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.125839] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.125877] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.125905] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221553137, support_ssl=false) [2025-05-15 09:40:39.125907] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.125939] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=31][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.125957] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.125965] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.125962] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553137, sess.client_addr_="172.22.0.1:39684") [2025-05-15 09:40:39.125986] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.125979] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553137, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.125998] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553136, proxy_sessid=0) [2025-05-15 09:40:39.126030] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=15][errcode=0] server close connection(sessid=3221553136, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.126044] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553136, proxy_sessid=0) [2025-05-15 09:40:39.126052] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.126060] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9085-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553136, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.126119] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221553136, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039125299, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.126130] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.126144] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553136, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.126156] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221553136, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039125299, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.126163] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.126173] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553136) [2025-05-15 09:40:39.126192] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.126186] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553136, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.126212] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.126227] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.126235] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.126264] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.126285] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553137, proxy_sessid=0) [2025-05-15 09:40:39.126323] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=13][errcode=0] server close connection(sessid=3221553137, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.126336] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553137, proxy_sessid=0) [2025-05-15 09:40:39.126343] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.126357] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9096-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553137, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.126407] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553137, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039125299, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.126429] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553137, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.126441] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553137, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039125299, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.126459] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553137) [2025-05-15 09:40:39.126483] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] connection close(sessid=3221553137, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.126627] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] new mysql sessid created(conn.sessid_=3221553138, support_ssl=false) [2025-05-15 09:40:39.126667] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553138, sess.client_addr_="172.22.0.1:39688") [2025-05-15 09:40:39.126681] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553138, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.126744] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553139, support_ssl=false) [2025-05-15 09:40:39.126793] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553139, sess.client_addr_="172.22.0.1:39692") [2025-05-15 09:40:39.126807] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553139, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.126854] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.126884] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.126905] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221553140, support_ssl=false) [2025-05-15 09:40:39.126902] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.126917] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.126924] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.126931] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.126941] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553140, sess.client_addr_="172.22.0.1:39700") [2025-05-15 09:40:39.126951] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.126953] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553140, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.126965] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553138, proxy_sessid=0) [2025-05-15 09:40:39.126983] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=7][errcode=0] server close connection(sessid=3221553138, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.126994] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553138, proxy_sessid=0) [2025-05-15 09:40:39.127001] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.127013] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C6-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553138, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.127035] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.127063] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.127057] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553138, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302039126358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.127078] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553138, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.127083] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.127090] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553138, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302039126358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.127101] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.127104] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553138) [2025-05-15 09:40:39.127117] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.127124] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.127122] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553138, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.127145] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.127161] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553139, proxy_sessid=0) [2025-05-15 09:40:39.127185] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=13][errcode=0] server close connection(sessid=3221553139, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.127193] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.127198] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553139, proxy_sessid=0) [2025-05-15 09:40:39.127205] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.127216] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.127211] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9086-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553139, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.127226] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.127240] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.127252] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.127259] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.127261] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553139, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039126358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.127276] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.127280] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553139, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.127291] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553140, proxy_sessid=0) [2025-05-15 09:40:39.127275] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] new mysql sessid created(conn.sessid_=3221553141, support_ssl=false) [2025-05-15 09:40:39.127290] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553139, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039126358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.127310] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553139) [2025-05-15 09:40:39.127318] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=16][errcode=0] server close connection(sessid=3221553140, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.127324] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553139, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.127334] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553140, proxy_sessid=0) [2025-05-15 09:40:39.127343] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:39.127350] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9097-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553140, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.127373] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] sm conn init succ(conn.sessid_=3221553141, sess.client_addr_="172.22.0.1:39714") [2025-05-15 09:40:39.127392] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553140, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039126358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.127397] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553141, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.127415] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553140, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.127427] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553140, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039126358, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.127476] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=48] mark session id unused(conn.sessid_=3221553140) [2025-05-15 09:40:39.127507] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553140, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.127628] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.127670] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.127689] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.127705] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.127713] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.127724] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.127737] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.127729] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=34] new mysql sessid created(conn.sessid_=3221553142, support_ssl=false) [2025-05-15 09:40:39.127759] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221553141, proxy_sessid=0) [2025-05-15 09:40:39.127780] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=8][errcode=0] server close connection(sessid=3221553141, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.127795] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553141, proxy_sessid=0) [2025-05-15 09:40:39.127801] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:39.127814] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C7-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553141, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.127828] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=36] sm conn init succ(conn.sessid_=3221553142, sess.client_addr_="172.22.0.1:39730") [2025-05-15 09:40:39.127848] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221553142, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.127871] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553143, support_ssl=false) [2025-05-15 09:40:39.127865] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553141, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.127896] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221553141, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.127915] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553141, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.127933] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553141) [2025-05-15 09:40:39.127928] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221553143, sess.client_addr_="172.22.0.1:39746") [2025-05-15 09:40:39.127960] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553143, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.127971] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] connection close(sessid=3221553141, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.128011] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221553144, support_ssl=false) [2025-05-15 09:40:39.128053] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553144, sess.client_addr_="172.22.0.1:39752") [2025-05-15 09:40:39.128051] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.128071] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553144, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128087] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039128040) [2025-05-15 09:40:39.128106] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.128112] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302038927989, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.128143] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.128153] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22] get wrs ts(ls_id={id:1}, delta=12341399583, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.128158] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.128161] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.128176] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12341227090, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.128176] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.128195] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.128197] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=21][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.128179] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.128205] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.128201] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.128208] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.128219] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128220] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=11][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.128218] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.128213] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.128231] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.128229] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553142, proxy_sessid=0) [2025-05-15 09:40:39.128232] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.128238] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.128242] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.128236] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.128255] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.128253] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.128252] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=12][errcode=0] server close connection(sessid=3221553142, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128256] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128257] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.128263] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.128266] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553142, proxy_sessid=0) [2025-05-15 09:40:39.128274] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:39.128268] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553143, proxy_sessid=0) [2025-05-15 09:40:39.128274] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039128158) [2025-05-15 09:40:39.128280] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.128283] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9087-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553142, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.128274] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.128298] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=15][errcode=0] server close connection(sessid=3221553143, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128300] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.128311] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=37][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.128314] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221553145, support_ssl=false) [2025-05-15 09:40:39.128313] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553143, proxy_sessid=0) [2025-05-15 09:40:39.128316] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.128320] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553146, support_ssl=false) [2025-05-15 09:40:39.128325] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:39.128324] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039128134) [2025-05-15 09:40:39.128327] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.128325] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221553142, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128343] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=16][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.128348] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553145, sess.client_addr_="172.22.0.1:39772") [2025-05-15 09:40:39.128349] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553142, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.128338] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9098-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553143, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.128364] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553146, sess.client_addr_="172.22.0.1:39758") [2025-05-15 09:40:39.128362] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553145, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128364] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221553142, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.128368] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=17] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128375] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553146, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128383] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553142) [2025-05-15 09:40:39.128387] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221553144, proxy_sessid=0) [2025-05-15 09:40:39.128396] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553143, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128408] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=12][errcode=0] server close connection(sessid=3221553144, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128417] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553143, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.128419] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553144, proxy_sessid=0) [2025-05-15 09:40:39.128425] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:39.128402] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553142, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.128424] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553143, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.128440] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553143) [2025-05-15 09:40:39.128431] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C8-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553144, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.128457] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553143, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.128616] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.128620] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.128617] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=35][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553144, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128641] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553144, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.128642] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.128645] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.128658] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.128660] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=30][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.128675] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.128676] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.128682] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.128684] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.128690] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.128691] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.128685] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553144, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039127418, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.128703] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553144) [2025-05-15 09:40:39.128691] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:39.128703] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128704] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128714] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553146, proxy_sessid=0) [2025-05-15 09:40:39.128716] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553145, proxy_sessid=0) [2025-05-15 09:40:39.128719] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=23] temporary file module use no blocks [2025-05-15 09:40:39.128720] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553144, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.128732] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=8][errcode=0] server close connection(sessid=3221553146, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128730] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:39.128735] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=8][errcode=0] server close connection(sessid=3221553145, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.128742] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553146, proxy_sessid=0) [2025-05-15 09:40:39.128744] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553145, proxy_sessid=0) [2025-05-15 09:40:39.128750] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:39.128752] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.128746] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:39.128757] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC9099-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553146, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.128761] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9088-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553145, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.128764] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11802019107, flush_io_finished_ret_:0, flush_io_finished_round_:1513, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:39.128811] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553145, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302039128478, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128813] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553146, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302039128478, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.128833] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553145, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.128841] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] kill and revert session(conn.sessid_=3221553146, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.128841] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553145, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302039128478, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.128853] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221553145) [2025-05-15 09:40:39.128853] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553146, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302039128478, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.128868] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553146) [2025-05-15 09:40:39.128874] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553145, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.128881] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553146, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.130062] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.140179] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.142812] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=30] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:39.150290] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.160401] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.170506] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.171382] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.171413] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5100327, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.171435] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=5100327, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.171450] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.171466] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=5100327, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.171482] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.171510] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.171527] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.171597] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=70][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.171607] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.171624] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.171644] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039171641, replica_locations:[]}) [2025-05-15 09:40:39.171665] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.171696] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.171708] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.171760] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.171826] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803062299, cache_obj->added_lc()=false, cache_obj->get_object_id()=15816, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.173251] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.173273] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.173289] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.173300] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.173310] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.173324] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.173340] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.173354] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.173368] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.173380] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.173396] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.173409] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039173409, replica_locations:[]}) [2025-05-15 09:40:39.173482] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=5098200, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=294, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.174205] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.174291] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.174348] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.174561] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.180497] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=29] ====== check clog disk timer task ====== [2025-05-15 09:40:39.180518] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=15] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:39.180619] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.182447] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:369, tid:4304}]) [2025-05-15 09:40:39.190760] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=48][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.191908] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=23] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:39.195959] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:39.195989] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:39.196008] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:39.196023] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:39.196036] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:39.196048] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:39.196063] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:39.196081] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=12][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:39.196091] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:39.196100] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:39.196107] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:39.196115] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:39.196123] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:39.196138] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:39.196157] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:39.196168] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.196180] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.196192] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:39.196205] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:39.196218] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:39.196227] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:39.196247] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:39.196268] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:39.196278] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:39.196286] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:39.196304] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:39.196318] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6471-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.196330] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:39.196344] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:39.196354] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:39.196364] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=8][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:39.196375] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302039195697, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:39.196388] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:39.196399] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:39.196501] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=12][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:39.196516] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:39.196528] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:39.196540] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:39.196552] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:39.196564] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=11][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:39.196574] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6471-0-0] [lt=9][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:39.200869] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.205536] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.206454] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.206500] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039206447) [2025-05-15 09:40:39.206517] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302039006388, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.206549] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12341454604, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.206567] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.206589] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.206605] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.206615] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.206632] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12341454604, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.206645] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.206657] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.206668] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.206677] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.206688] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.206700] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.206712] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039206530) [2025-05-15 09:40:39.210976] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.218264] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.218288] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.218303] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.218317] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.218327] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.218339] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.218347] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.218358] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.218369] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302039218246, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.218381] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.218389] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.218442] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.218454] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.218464] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.218474] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.221087] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.223510] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.223534] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.223619] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=84][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.224711] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=1][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:39.224734] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.224751] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.224765] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:39.224776] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:39.224788] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:39.228113] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12341500195, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.228136] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.228157] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.228171] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.228180] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.228193] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.228202] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.228209] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039228100) [2025-05-15 09:40:39.228221] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.228245] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039228217) [2025-05-15 09:40:39.228258] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039028174, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.228280] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341327702, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.228295] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.228310] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.228322] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.228331] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.228344] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.228358] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.228366] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039228269) [2025-05-15 09:40:39.231197] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.235415] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:39.235435] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.235446] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.237433] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4580][T1002_TimerWK3_TenantMetaMemMg][T1002][Y0-0000000000000000-0-0] [lt=21] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:728, free_obj_hold_cnt:0, allocator used:2795520, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=3002880, tablets_mem_limit=7774753240, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:9, free_obj_hold_cnt:0, allocator used:18432, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=728) [2025-05-15 09:40:39.241305] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.243437] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-05-15 09:40:39.251416] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.255914] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.255935] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:39.255946] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:39.255963] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:39.255976] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:39.256003] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:39.259357] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:39.259384] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:39.259393] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:39.259412] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:39.259429] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=18] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11802149776, flush_io_finished_ret_:0, flush_io_finished_round_:1513, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:39.261523] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.269408] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=26] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:39.271980] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.273601] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.273623] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4998097, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.273640] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4998097, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.273654] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.273662] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4998097, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.273676] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.273691] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.273707] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.273721] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.273733] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.273748] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.273766] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039273765, replica_locations:[]}) [2025-05-15 09:40:39.273784] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.273803] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:294, local_retry_times:294, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:39.273825] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.273836] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.273852] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:39.273862] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:39.273872] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:39.273889] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:39.273904] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.273952] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803164429, cache_obj->added_lc()=false, cache_obj->get_object_id()=15817, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.275037] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.275064] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.275097] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.275112] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.275127] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.275140] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.275151] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.275165] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.275180] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.275195] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.275205] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.275216] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.275232] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.275246] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039275245, replica_locations:[]}) [2025-05-15 09:40:39.275263] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.275276] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.275289] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.275305] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:39.275316] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:39.275327] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:39.275346] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:39.275361] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:39.275368] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:39.275381] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:39.275391] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:39.275401] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:39.275414] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:39.275426] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:39.275437] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:39.275447] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:39.275458] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:39.275469] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:39.275480] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:39.275510] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:39.275523] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.275535] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:39.275546] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:39.275559] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:39.275567] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=295, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:39.275591] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=4996090, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=295, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.282261] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6452-0-0] [lt=20][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.282285] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6452-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.282310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6452-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.282329] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6452-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.282344] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6452-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.282360] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:39.282375] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.282390] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.282404] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302039282237, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:39.282421] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.282434] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:39.282517] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1001, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:39.282564] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=47][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:39.282639] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.283790] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6453-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.283820] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6453-0-0] [lt=30][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.283836] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6453-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.283852] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6453-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.283865] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6453-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.283881] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:39.283894] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.283907] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.283918] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302039283781, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:39.283934] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.283947] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:39.283993] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] fail to select for update snapshot_gc_scn(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001) [2025-05-15 09:40:39.284008] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] failed to load updated info(ret=-5627) [2025-05-15 09:40:39.284020] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to try update info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:39.291464] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=22][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:39.291484] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.291508] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.291524] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.291539] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.291547] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.291560] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.291575] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.291585] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.291600] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.291611] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.291626] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.291657] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.291671] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=12] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:39.292751] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.293250] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:39.293276] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=23][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:39.293291] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:39.293307] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:39.293321] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:39.293332] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:39.293348] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:39.293359] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:39.293367] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=6][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:39.293374] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=6][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:39.293384] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:39.293396] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:39.293407] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:39.293418] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:39.293437] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:39.293449] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.293460] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.293471] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:39.293480] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:39.293500] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=18][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:39.293506] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:39.293522] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=12][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:39.293541] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:39.293551] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:39.293557] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:39.293573] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:39.293586] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.293597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:39.293609] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:39.293621] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:39.293633] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:39.293645] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302039293082, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:39.293658] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:39.293668] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=8][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:39.293748] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=12][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:39.293761] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=12][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:39.293773] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:39.293784] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:39.293795] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:39.293804] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=8][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:39.293814] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6391-0-0] [lt=10][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:39.302859] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.305596] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.306534] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12341555181, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.306564] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.306591] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.306613] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.306628] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.306649] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1001}, delta=12341555181, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.306665] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.306683] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.306698] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.306708] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.306727] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.306738] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.306751] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039306507) [2025-05-15 09:40:39.308073] WDIAG load_file_to_string (utility.h:673) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=30][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:39.308093] WDIAG get_ethernet_speed (utility.cpp:575) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:39.308104] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:39.308121] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:39.312965] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.318299] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.318321] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.318335] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.318360] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.318370] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.318381] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.318390] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.318399] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.318409] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302039318284, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.318421] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.318432] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.318484] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.318507] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.318518] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.318528] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.323071] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.323572] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.323596] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.323613] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.324769] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:39.324792] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.324808] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.324821] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:39.324835] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:39.324847] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:39.328167] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.328195] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039328161) [2025-05-15 09:40:39.328212] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302039128133, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.328237] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341599704, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.328254] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.328276] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.328288] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.328280] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.328298] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.328298] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039328277) [2025-05-15 09:40:39.328309] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.328309] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039228269, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.328319] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.328326] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039328225) [2025-05-15 09:40:39.328334] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12341427211, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.328348] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.328363] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.328376] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.328387] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.328401] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.328410] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.328416] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039328324) [2025-05-15 09:40:39.333182] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.335474] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:39.335503] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.335516] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.343075] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:39.343293] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.353409] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.355080] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-05-15 09:40:39.355112] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.355142] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:39.355152] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:39.355162] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:39.355250] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=88][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:39.355260] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.355300] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=40][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:39.355307] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:39.355316] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:39.355366] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:39.363524] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.366000] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:39.366071] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:39.366137] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:39.366198] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:39.366227] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:4814, total:4823) [2025-05-15 09:40:39.366261] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:39.366313] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=46] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:4814, total:4823) [2025-05-15 09:40:39.366342] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=8] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:39.366359] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:39.366372] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:39.366392] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628774][2nd=1901]}) [2025-05-15 09:40:39.373629] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.375715] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.375739] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4895983, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.375764] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4895983, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.375875] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=112][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.375885] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4895983, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.375912] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.375928] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.375943] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.375957] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.375969] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.375985] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.376000] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039375998, replica_locations:[]}) [2025-05-15 09:40:39.376018] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.376038] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:295, local_retry_times:295, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:39.376057] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.376068] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.376084] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:39.376093] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:39.376099] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:39.376133] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:39.376146] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.376187] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803266666, cache_obj->added_lc()=false, cache_obj->get_object_id()=15818, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.377313] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.377342] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.377374] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.377390] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.377404] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.377418] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.377426] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.377439] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.377454] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.377468] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.377481] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.377502] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.377514] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.377529] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039377528, replica_locations:[]}) [2025-05-15 09:40:39.377546] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.377559] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.377572] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.377588] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:39.377599] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:39.377611] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:39.377628] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:39.377642] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:39.377654] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:39.377664] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:39.377674] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:39.377682] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:39.377696] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:39.377708] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:39.377716] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:39.377727] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:39.377737] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:39.377748] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:39.377759] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:39.377778] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:39.377789] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.377800] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:39.377811] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:39.377823] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:39.377832] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=296, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:39.377856] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=4893826, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=296, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.383737] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.393841] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.395000] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.395881] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.395937] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=16] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.395975] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.396049] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.396559] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.396992] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:39.403946] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.405655] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.406576] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001) [2025-05-15 09:40:39.406616] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=36][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.406644] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039406567) [2025-05-15 09:40:39.406660] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302039206529, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.406694] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12341654673, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.406713] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.406733] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.406745] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.406755] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.406771] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12341654673, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.406790] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.406803] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.406814] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.406823] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.406835] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.406844] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.406853] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039406677) [2025-05-15 09:40:39.414054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.414374] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:39.414397] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.414409] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:39.414433] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:39.418511] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.418534] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.418560] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.418580] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.418594] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.418606] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.418619] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.418629] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.418640] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302039418469, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.418656] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.418665] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.418742] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.418752] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.418762] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.418773] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.423629] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.423650] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.423666] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.424159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.424827] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:39.424850] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.424865] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.424881] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:39.424893] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:39.424904] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:39.428237] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12341700250, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.428257] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.428279] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.428307] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.428319] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.428340] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.428362] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039428336) [2025-05-15 09:40:39.428376] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039328319, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.428398] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341527757, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.428411] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.428427] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.434273] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.435532] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=101][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:39.435550] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.435560] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.444380] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.446430] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] Cache replace map node details(ret=0, replace_node_count=0, replace_time=2904, replace_start_pos=6100000, replace_num=100000) [2025-05-15 09:40:39.446454] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=10) [2025-05-15 09:40:39.455258] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.455281] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.455293] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:39.455319] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:39.455492] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.465607] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.475715] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.477973] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.477995] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4793723, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.478011] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4793723, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.478025] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.478034] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4793723, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.478048] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.478064] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.478079] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.478094] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.478106] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.478122] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.478137] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039478136, replica_locations:[]}) [2025-05-15 09:40:39.478156] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.478174] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:296, local_retry_times:296, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:39.478193] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.478205] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.478221] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:39.478234] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:39.478240] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:39.478258] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:39.478271] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.478318] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803368795, cache_obj->added_lc()=false, cache_obj->get_object_id()=15819, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.479378] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.479405] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.479435] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.479451] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.479466] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.479476] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.479501] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.479511] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.479524] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.479537] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.479550] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.479559] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.479574] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.479586] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039479585, replica_locations:[]}) [2025-05-15 09:40:39.479602] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.479613] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.479625] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:39.479647] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:39.479658] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:39.479669] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:39.479687] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:39.479702] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:39.479710] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:39.479721] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:39.479731] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:39.479741] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:39.479754] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:39.479766] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:39.479777] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:39.479789] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:39.479799] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:39.479811] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:39.479821] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:39.479836] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:39.479848] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:39.479857] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:39.479868] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:39.479877] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:39.479887] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=297, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:39.479910] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=4791771, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=297, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.485821] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.495926] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.505719] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.506031] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.506542] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:39.506567] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.506646] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12341755216, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.506654] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=87][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:39.506674] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:39.506668] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.506685] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:39.506688] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.506694] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:39.506702] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:39.506709] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1001}, delta=12341755216, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.506715] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:39.506705] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=41][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.506722] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.506726] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.506739] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:39.506754] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:39.506767] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:39.506780] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:39.506800] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:39.506816] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:39.506854] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=1] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:39.510843] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C666A-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.510861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C666A-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.510884] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C666A-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.510899] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291F9C666A-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.510911] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666A-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.510922] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:39.510933] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.510943] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.510953] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302039510821, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:39.510965] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.510975] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.512242] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.512261] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=19][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.512271] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:39.512278] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=7][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:39.512287] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=9][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:39.512311] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=8][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:39.512321] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.512337] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_FreInfoReload][T1002][YB427F000001-000635291F9C666B-0-0] [lt=15][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5157, ret=-5157) [2025-05-15 09:40:39.512347] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:39.512357] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:39.512365] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:39.512375] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78ca3b0, start=1747302039512233, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:39.512385] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:39.512395] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] execute sql failed(ret=-5157, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:39.512432] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to select for update snapshot_gc_scn(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.512444] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] failed to load updated info(ret=-5157) [2025-05-15 09:40:39.512453] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4579][T1002_TimerWK2_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to try update info(tmp_ret=-5157, tmp_ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:39.516139] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.518412] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.518443] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.523688] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.523710] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.523721] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.524887] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:39.524910] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.524921] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.524937] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:39.524948] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:39.524960] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:39.526250] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.528286] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.528320] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039528281) [2025-05-15 09:40:39.528336] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302039328224, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.528365] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341799738, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.528382] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.528396] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.528426] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039528392) [2025-05-15 09:40:39.528439] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039428387, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.528461] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341627245, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.528475] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.535591] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:39.535611] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.535621] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.536358] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.542333] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:39.543339] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=25] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:39.543657] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:39.546469] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.556592] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.566702] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.569172] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.569196] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.569223] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:39.569236] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.569274] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=31] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:39.569289] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:39.569310] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:39.569323] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:39.569338] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:39.569701] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:39.569713] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:39.569725] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:39.576812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.577594] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6694-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.580032] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.580056] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4691667, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.580070] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4691667, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.580121] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=51][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.580130] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4691667, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.580144] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.580160] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.580175] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.580186] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.580197] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.580212] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.580228] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039580227, replica_locations:[]}) [2025-05-15 09:40:39.580247] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.580266] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.580286] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.580298] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.580345] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.580390] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803470866, cache_obj->added_lc()=false, cache_obj->get_object_id()=15820, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.581470] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.581504] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.581520] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.581531] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.581542] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.581555] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.581570] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.581584] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.581598] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.581609] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.581625] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.581639] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039581639, replica_locations:[]}) [2025-05-15 09:40:39.581700] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=4689982, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=298, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.583287] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=26][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:31, tid:4630}]) [2025-05-15 09:40:39.583770] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553158, support_ssl=false) [2025-05-15 09:40:39.583878] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=41] sm conn init succ(conn.sessid_=3221553158, sess.client_addr_="172.22.0.1:39782") [2025-05-15 09:40:39.583901] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553158, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.584165] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.584209] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.584248] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.584278] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.584297] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.584307] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.584337] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.584358] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553158, proxy_sessid=0) [2025-05-15 09:40:39.584401] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=15][errcode=0] server close connection(sessid=3221553158, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.584415] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553158, proxy_sessid=0) [2025-05-15 09:40:39.584424] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.584438] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90C9-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553158, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.584510] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553158, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039583674, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.584539] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221553158, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.584549] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553158, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039583674, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.584569] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553158) [2025-05-15 09:40:39.584586] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553158, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.586939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.596539] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=72791040) [2025-05-15 09:40:39.596593] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:39.597060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.605785] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.606704] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.606744] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039606694) [2025-05-15 09:40:39.606765] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302039406671, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.606832] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=40] get wrs ts(ls_id={id:1}, delta=12341854653, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.606854] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.606881] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.606900] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.606913] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.606931] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12341855711, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.606944] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.606959] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.606971] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.606982] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.606996] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.607008] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.607017] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039606805) [2025-05-15 09:40:39.607195] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.618667] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.619422] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.623754] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.623781] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.623800] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.624956] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.625002] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302039624905, register_time_us=-1, curr_time_us=1747302039624999) [2025-05-15 09:40:39.625033] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:39.625062] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] re_register_parent reason(palf_id=1001, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302039624905, register_time_us=-1, curr_time_us=1747302039625062) [2025-05-15 09:40:39.625084] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1001, self="127.0.0.1:2882") [2025-05-15 09:40:39.627123] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553160, support_ssl=false) [2025-05-15 09:40:39.627243] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221553160, sess.client_addr_="172.22.0.1:39794") [2025-05-15 09:40:39.627261] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553160, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.627278] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553161, support_ssl=false) [2025-05-15 09:40:39.627334] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553161, sess.client_addr_="172.22.0.1:39796") [2025-05-15 09:40:39.627352] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553161, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.627505] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.627545] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.627584] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=14][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.627608] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.627616] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.627627] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.627636] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.627649] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.627668] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.627680] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.627691] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553160, proxy_sessid=0) [2025-05-15 09:40:39.627710] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.627721] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.627730] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.627732] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=15][errcode=0] server close connection(sessid=3221553160, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.627744] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553160, proxy_sessid=0) [2025-05-15 09:40:39.627752] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.627756] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.627775] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553161, proxy_sessid=0) [2025-05-15 09:40:39.627767] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909A-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553160, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.627812] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=15][errcode=0] server close connection(sessid=3221553161, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.627826] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553161, proxy_sessid=0) [2025-05-15 09:40:39.627835] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.627830] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553160, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039627021, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.627845] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9089-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553161, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.627856] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553160, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.627869] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553160, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039627021, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.627884] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221553160) [2025-05-15 09:40:39.627898] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553161, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039627021, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.627898] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553160, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.627921] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553161, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.627929] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553161, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039627021, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.627947] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553161) [2025-05-15 09:40:39.627959] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553161, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.628216] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553162, support_ssl=false) [2025-05-15 09:40:39.628256] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221553162, sess.client_addr_="172.22.0.1:39808") [2025-05-15 09:40:39.628272] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553162, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.628363] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=12341900243, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.628386] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.628411] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.628430] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.628438] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.628440] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.628439] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12341727750, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.628459] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.628457] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.628467] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.628467] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.628472] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.628476] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039628344) [2025-05-15 09:40:39.628485] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.628505] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.628499] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.628515] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.628516] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.628525] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.628527] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.628535] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.628533] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039628423) [2025-05-15 09:40:39.628553] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.628564] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553163, support_ssl=false) [2025-05-15 09:40:39.628569] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553162, proxy_sessid=0) [2025-05-15 09:40:39.628589] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=8][errcode=0] server close connection(sessid=3221553162, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.628601] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553162, proxy_sessid=0) [2025-05-15 09:40:39.628597] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] new mysql sessid created(conn.sessid_=3221553164, support_ssl=false) [2025-05-15 09:40:39.628609] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:39.628607] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553163, sess.client_addr_="172.22.0.1:39824") [2025-05-15 09:40:39.628617] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CA-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553162, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.628621] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553163, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.628633] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553164, sess.client_addr_="172.22.0.1:39834") [2025-05-15 09:40:39.628648] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553164, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.628686] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553162, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039628078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.628707] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553162, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.628714] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553162, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039628078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.628731] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553162) [2025-05-15 09:40:39.628743] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553162, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.628790] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.628819] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.628835] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.628850] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.628857] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.628864] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.628881] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.628885] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.628895] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553163, proxy_sessid=0) [2025-05-15 09:40:39.628914] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=9][errcode=0] server close connection(sessid=3221553163, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.628915] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.628926] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553163, proxy_sessid=0) [2025-05-15 09:40:39.628934] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.628928] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.628942] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.628941] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909B-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553163, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.628957] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.628963] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.628974] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=6] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.628991] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553164, proxy_sessid=0) [2025-05-15 09:40:39.628990] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553163, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039628078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.629009] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=8][errcode=0] server close connection(sessid=3221553164, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.629015] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553163, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.629021] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553164, proxy_sessid=0) [2025-05-15 09:40:39.629029] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:39.629029] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553163, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302039628078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.629045] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553163) [2025-05-15 09:40:39.629038] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908A-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553164, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.629058] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553163, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.629087] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553164, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302039628078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.629113] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553164, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.629121] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553164, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302039628078, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.629150] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] mark session id unused(conn.sessid_=3221553164) [2025-05-15 09:40:39.629165] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553164, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.629213] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553165, support_ssl=false) [2025-05-15 09:40:39.629252] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553165, sess.client_addr_="172.22.0.1:39840") [2025-05-15 09:40:39.629268] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553165, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.629413] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553166, support_ssl=false) [2025-05-15 09:40:39.629445] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.629471] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.629500] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.629521] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.629530] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.629550] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=19][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.629552] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.629560] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221553166, sess.client_addr_="172.22.0.1:39854") [2025-05-15 09:40:39.629573] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.629590] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553165, proxy_sessid=0) [2025-05-15 09:40:39.629580] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553166, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.629615] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221553167, support_ssl=false) [2025-05-15 09:40:39.629614] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=10][errcode=0] server close connection(sessid=3221553165, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.629642] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=27][errcode=-4018] get session fail(ret=-4018, sessid=3221553165, proxy_sessid=0) [2025-05-15 09:40:39.629662] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=20][errcode=-4016] session is null [2025-05-15 09:40:39.629666] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553167, sess.client_addr_="172.22.0.1:39870") [2025-05-15 09:40:39.629669] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CB-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553165, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.629692] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553167, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.629728] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553165, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039629143, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.629754] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553165, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.629763] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553165, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039629143, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.629783] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553165) [2025-05-15 09:40:39.629784] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=41] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.629796] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553165, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.629832] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.629846] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.629860] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.629873] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.629879] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.629891] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.629889] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] new mysql sessid created(conn.sessid_=3221553168, support_ssl=false) [2025-05-15 09:40:39.629901] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553166, proxy_sessid=0) [2025-05-15 09:40:39.629922] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=11][errcode=0] server close connection(sessid=3221553166, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.629927] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553168, sess.client_addr_="172.22.0.1:39874") [2025-05-15 09:40:39.629934] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553166, proxy_sessid=0) [2025-05-15 09:40:39.629943] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:39.629944] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553168, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.629949] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909C-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553166, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.629982] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.629993] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553166, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039629143, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.630009] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.630018] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553166, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.630023] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.630032] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553166, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039629143, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.630041] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.630050] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.630050] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553166) [2025-05-15 09:40:39.630057] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.630075] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630074] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553166, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.630091] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553167, proxy_sessid=0) [2025-05-15 09:40:39.630110] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=8][errcode=0] server close connection(sessid=3221553167, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630113] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553169, support_ssl=false) [2025-05-15 09:40:39.630123] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553167, proxy_sessid=0) [2025-05-15 09:40:39.630131] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:39.630139] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908B-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553167, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.630145] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553169, sess.client_addr_="172.22.0.1:39880") [2025-05-15 09:40:39.630161] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553169, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.630183] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553167, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302039629143, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.630206] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553167, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.630215] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553167, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302039629143, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.630233] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553167) [2025-05-15 09:40:39.630201] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=42] new mysql sessid created(conn.sessid_=3221553170, support_ssl=false) [2025-05-15 09:40:39.630246] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553167, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.630274] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=48] sm conn init succ(conn.sessid_=3221553170, sess.client_addr_="172.22.0.1:39886") [2025-05-15 09:40:39.630286] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.630300] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553170, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.630321] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=17] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.630344] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.630364] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.630363] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.630378] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.630384] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.630398] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.630401] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630414] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.630420] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553168, proxy_sessid=0) [2025-05-15 09:40:39.630431] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.630440] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.630447] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=12][errcode=0] server close connection(sessid=3221553168, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630458] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=18][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.630461] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553168, proxy_sessid=0) [2025-05-15 09:40:39.630471] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:39.630474] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630503] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=26][errcode=-4018] get session fail(ret=-4018, sessid=3221553169, proxy_sessid=0) [2025-05-15 09:40:39.630501] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CC-0-0] [lt=28] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553168, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.630532] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=15][errcode=0] server close connection(sessid=3221553169, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630546] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553169, proxy_sessid=0) [2025-05-15 09:40:39.630565] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=19][errcode=-4016] session is null [2025-05-15 09:40:39.630585] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=34] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:39.630577] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909D-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553169, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.630609] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553168, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039630201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.630639] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221553168, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.630633] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553169, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302039630201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.630644] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:39.630654] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553169, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.630648] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553168, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302039630201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.630665] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553168) [2025-05-15 09:40:39.630665] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553169, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302039630201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.630678] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553169) [2025-05-15 09:40:39.630671] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:39.630678] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553168, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.630687] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:39.630692] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553169, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.630705] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:39.630712] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:39.630723] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=6] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630739] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553170, proxy_sessid=0) [2025-05-15 09:40:39.630756] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=7][errcode=0] server close connection(sessid=3221553170, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.630770] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553170, proxy_sessid=0) [2025-05-15 09:40:39.630776] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:39.630783] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908C-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553170, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:39.630834] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553170, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039630201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:39.630868] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221553170, proxy_sessid=0, ret=0) [2025-05-15 09:40:39.630879] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553170, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302039630201, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:39.630901] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553170) [2025-05-15 09:40:39.630920] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553170, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:39.635648] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.639707] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.646540] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=12] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=9) [2025-05-15 09:40:39.647077] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=20] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4839939501, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:39.647382] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=25] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4839939515, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:39.649824] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.659938] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.659979] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4050][LSUserTblUp6][T0][YB427F000001-000635291D6D1664-0-0] [lt=37][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001, task={tenant_id:1002, ls_id:{id:1}, add_timestamp:1747300526212290}) [2025-05-15 09:40:39.670063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.680198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.681865] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.681908] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=35][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4589847, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.681936] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4589847, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.681956] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.681975] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4589847, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.681994] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.682018] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.682040] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.682060] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.682074] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.682097] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.682120] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039682117, replica_locations:[]}) [2025-05-15 09:40:39.682141] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.682176] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.682189] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.682229] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.682296] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803572767, cache_obj->added_lc()=false, cache_obj->get_object_id()=15821, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.683534] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:48, tid:4602}]) [2025-05-15 09:40:39.683754] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.683779] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.683800] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.683818] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.683829] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.683847] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.683872] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.683896] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.683905] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.683921] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.683934] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039683934, replica_locations:[]}) [2025-05-15 09:40:39.684009] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=4587673, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=299, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.690310] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.700423] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.705844] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.706767] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12341955173, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.706793] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.706812] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.706829] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.706840] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.706857] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12341955173, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.706871] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.706886] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.706921] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=35][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.706928] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.706940] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.706957] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.706964] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039706750) [2025-05-15 09:40:39.710535] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.718647] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.718670] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.718687] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.718708] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.718723] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.718738] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.718749] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.718761] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302039718626, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.718775] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.718786] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.718853] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.718867] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.718879] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.718891] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.720646] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.723812] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.723833] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.723845] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.728410] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.728436] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039728404) [2025-05-15 09:40:39.728455] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302039528348, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.728481] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12341999702, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.728483] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.728512] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039728479) [2025-05-15 09:40:39.728514] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.728523] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039528451, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.728536] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.728544] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12341827209, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.728547] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.728557] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.728557] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.728568] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.728574] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.728577] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.728587] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.728587] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039728470) [2025-05-15 09:40:39.728597] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.728608] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.728617] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.728623] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039728534) [2025-05-15 09:40:39.732325] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.741973] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3954][TimerWK1_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=21] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:39.742442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.743624] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=10] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44870239027, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:39.752568] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.756065] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.756086] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:39.756096] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:39.756114] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:39.756129] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=12478464) [2025-05-15 09:40:39.756157] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=28, size_used=0, mem_used=0) [2025-05-15 09:40:39.756187] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:39.762689] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.772812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.773982] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4336][T1_ArcSender][T1][YB427F000001-000635291B9C6112-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:39.782920] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.784175] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.784204] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4487538, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.784259] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=55][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4487538, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.784299] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.784318] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.784330] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.784348] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.784371] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039784368, replica_locations:[]}) [2025-05-15 09:40:39.784392] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.784422] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.785955] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.785976] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.785992] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.786013] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.786026] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.786036] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.786051] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.786064] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039786064, replica_locations:[]}) [2025-05-15 09:40:39.786135] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=4485547, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=300, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.793036] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.799018] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [4289][T1_TimerWK0_PalfGC][T1][Y0-0000000000000000-0-0] [lt=1] BlockGCTimerTask success(ret=0, cost_time_us=7, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"127.0.0.1:2882", log_dir:"/root/ob/observer/store/clog/tenant_1", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-15 09:40:39.803176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=31][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.805868] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.806815] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.806852] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039806809) [2025-05-15 09:40:39.806867] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302039606805, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.806896] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12342055700, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.806912] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.806933] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.806949] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.806961] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.806978] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12342055700, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.806991] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.807006] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.807018] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.807029] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.807040] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.807050] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.807056] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039806879) [2025-05-15 09:40:39.813288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.818675] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.818696] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.818717] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.818734] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.818747] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.818761] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.818775] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.818786] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302039818654, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.818799] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.818810] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.818870] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.818881] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.818893] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.818906] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.820185] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:39.823400] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.823872] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.823893] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.823910] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.828479] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12342100233, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.828511] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.828533] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.828548] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.828542] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.828560] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.828561] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039828539) [2025-05-15 09:40:39.828571] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.828571] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039728534, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.828583] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.828589] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039828464) [2025-05-15 09:40:39.828600] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12341927740, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.828615] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.828631] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.828645] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.828656] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.828666] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.828676] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.828682] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039828591) [2025-05-15 09:40:39.833509] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.839713] INFO [STORAGE] diagnose_for_suggestion (ob_compaction_suggestion.cpp:456) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=9] [COMPACTION DAG STATUS] (start_time=1747302029838844, end_time=1747302039839709, dag_status={}) [2025-05-15 09:40:39.839737] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=23] dump_dag_status(priority="PRIO_COMPACTION_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839753] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=16] dump_dag_status(priority="PRIO_HA_HIGH", limits=8, running_task=0, adaptive_task_limit=8, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839764] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_COMPACTION_MID", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839775] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_HA_MID", limits=5, running_task=0, adaptive_task_limit=5, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839786] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status(priority="PRIO_COMPACTION_LOW", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839797] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_HA_LOW", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839810] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=12] dump_dag_status(priority="PRIO_DDL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839824] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=14] dump_dag_status(priority="PRIO_DDL_HIGH", limits=6, running_task=0, adaptive_task_limit=6, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839835] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:2926) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(priority="PRIO_TTL", limits=2, running_task=0, adaptive_task_limit=2, ready_dag_count=0, waiting_dag_count=0, rank_dag_count=0) [2025-05-15 09:40:39.839850] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:3745) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=11] dump_dag_status[DAG_NET](running_dag_net_map_size=0, blocking_dag_net_list_size=0, running_dag_net_list_size=0) [2025-05-15 09:40:39.839860] INFO [COMMON] dump_dag_status (ob_tenant_dag_scheduler.cpp:4494) [4280][T1_DagScheduler][T1][Y0-0000000000000000-0-0] [lt=10] dump_dag_status(dag_cnt=0, total_worker_cnt_=0, total_running_task_cnt=0, work_thread_num_=43, scheduled_task_cnt=0) [2025-05-15 09:40:39.843617] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.846631] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:39.853730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.860072] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=35] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:39.863838] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.873944] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.884054] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.886260] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.886286] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4385436, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.886303] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4385436, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.886327] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.886341] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.886353] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.886368] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.886383] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039886382, replica_locations:[]}) [2025-05-15 09:40:39.886401] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.886425] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.887622] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.887644] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.887660] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.887682] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.887696] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.887704] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.887719] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.887733] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039887733, replica_locations:[]}) [2025-05-15 09:40:39.887796] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=4383886, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=301, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.894165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.904274] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.904402] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4462][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=12] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:39.905923] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:39.906885] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12342155210, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.906908] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.906928] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.906945] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.906956] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.906972] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12342155210, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:39.906986] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.907000] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:39.907009] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:39.907019] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:39.907029] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.907039] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:39.907045] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039906870) [2025-05-15 09:40:39.914383] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.914518] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4460][T1001_TimerWK0_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.918738] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.918756] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.918774] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.918792] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.918806] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.918819] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.918831] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.918844] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302039918719, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.918853] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.918864] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:39.918926] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:39.918937] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:39.918944] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:39.918956] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:39.920715] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302039920712, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007614, add_bytes: 0 [2025-05-15 09:40:39.923930] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:39.923949] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:39.923962] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:39.924496] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.926599] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=27] PNIO [ratelimit] time: 1747302039926597, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:39.927309] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:39.927314] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=36] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:39.928526] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.928550] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039928521) [2025-05-15 09:40:39.928567] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302039728470, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.928592] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12342199743, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:39.928601] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:39.928608] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.928620] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302039928598) [2025-05-15 09:40:39.928625] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:39.928634] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039828587, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:39.928638] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.928649] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.928656] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12342028308, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:39.928660] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.928668] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:39.928675] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039928582) [2025-05-15 09:40:39.928671] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:39.928687] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:39.928697] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:39.928708] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:39.928723] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.928732] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:39.928739] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302039928646) [2025-05-15 09:40:39.934608] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.936387] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=17] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4841029911, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:39.936418] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:39.936428] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1aaefec3a0) [2025-05-15 09:40:39.936440] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4841029911, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:39.943879] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=24] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:39.944720] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.954823] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.959374] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=35] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:39.959402] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:39.959416] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=13] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:39.959434] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:39.959450] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11802849793, flush_io_finished_ret_:0, flush_io_finished_round_:1514, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:39.960437] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:39.961428] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=25] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:39.964928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.975032] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.984136] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:26, tid:3961}]) [2025-05-15 09:40:39.985137] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.987917] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.987941] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4283780, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.987956] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4283780, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.987970] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.987981] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4283780, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.987995] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.988011] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.988025] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.988038] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.988049] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.988061] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.988074] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039988073, replica_locations:[]}) [2025-05-15 09:40:39.988091] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:39.988114] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:39.988125] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:39.988164] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.988213] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803878689, cache_obj->added_lc()=false, cache_obj->get_object_id()=15824, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:39.989328] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:39.989349] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.989364] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:39.989377] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:39.989387] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:39.989400] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:39.989412] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:39.989426] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.989444] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:39.989458] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:39.989471] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:39.989484] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302039989483, replica_locations:[]}) [2025-05-15 09:40:39.989551] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=4282130, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=302, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:39.995246] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:39.995869] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE6-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.995888] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE6-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.995909] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE6-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.995925] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE6-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.995936] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE6-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.995948] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:39.995957] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.995968] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.995978] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302039995846, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:39.995991] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.996001] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:39.996076] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:39.997314] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE7-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.997331] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE7-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:39.997342] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE7-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:39.997354] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE7-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:39.997364] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE7-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:39.997374] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:39.997384] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:39.997391] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:39.997400] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302039997305, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:39.997410] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:39.997419] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:39.997455] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:39.998864] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:40.005360] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.005985] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.006905] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.006925] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.006934] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:40.006932] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.006956] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040006926) [2025-05-15 09:40:40.006968] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302039806879, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.006973] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=39][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:40.006980] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:40.006989] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:40.006996] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12342255663, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.007006] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:40.007011] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.007021] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:40.007041] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.007056] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.007056] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:40.007068] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.007083] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12342255663, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.007097] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.007120] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.007132] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.007139] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.007153] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.007164] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.007171] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040006979) [2025-05-15 09:40:40.011097] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] ====== tenant freeze timer task ====== [2025-05-15 09:40:40.015468] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.018865] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.018885] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.018908] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.018924] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.018937] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.018950] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.018962] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.018973] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.018985] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302040018841, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.018998] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.019009] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.019073] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.019087] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.019097] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.019107] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.023989] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.024010] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.024026] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.025587] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.028586] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:40.028622] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=12342300194, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.028640] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.028659] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:40.028661] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.028674] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.028683] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.028688] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040028657) [2025-05-15 09:40:40.028698] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.028701] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302039928646, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.028710] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.028716] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040028609) [2025-05-15 09:40:40.028721] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12342127701, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.028733] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.028750] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:40.028764] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.028775] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.028789] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.028798] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.028804] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040028712) [2025-05-15 09:40:40.035696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.045800] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.046722] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:40.055909] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.066015] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.066079] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=17] ====== check clog disk timer task ====== [2025-05-15 09:40:40.066100] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=18] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:40.070149] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:40.070162] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC96AF-0-0] [lt=26][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302040070133}) [2025-05-15 09:40:40.070189] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:40.070196] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:40.070207] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:40.070221] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.070225] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:40.070237] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:40.070252] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:40.070267] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:40.070362] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=96][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:40.070632] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.070646] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:40.070659] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:40.076125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.085113] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553177, support_ssl=false) [2025-05-15 09:40:40.085232] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] sm conn init succ(conn.sessid_=3221553177, sess.client_addr_="172.22.0.1:39888") [2025-05-15 09:40:40.085253] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553177, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.085409] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=37] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.085453] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.085482] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.085512] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.085527] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.085536] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.085654] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=103] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.085673] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553177, proxy_sessid=0) [2025-05-15 09:40:40.085710] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=10][errcode=0] server close connection(sessid=3221553177, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.085723] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553177, proxy_sessid=0) [2025-05-15 09:40:40.085732] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.085747] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CD-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553177, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.085803] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553177, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040085129, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.085828] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553177, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.085838] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553177, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040085129, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.085856] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553177) [2025-05-15 09:40:40.085877] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553177, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.086238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.089682] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.089707] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4182017, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.089724] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4182017, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.089735] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.089747] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4182017, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.089762] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.089787] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.089803] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.089816] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.089826] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.089841] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.089857] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040089855, replica_locations:[]}) [2025-05-15 09:40:40.089873] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.089896] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.089908] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:40.089931] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.089975] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11803980454, cache_obj->added_lc()=false, cache_obj->get_object_id()=15825, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.091238] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.091260] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.091275] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.091288] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.091298] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.091312] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.091325] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.091339] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.091351] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.091362] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.091375] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.091387] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040091387, replica_locations:[]}) [2025-05-15 09:40:40.091454] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=4180228, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=303, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.096342] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.101690] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=38] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:40.101724] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=25] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:40.106043] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.106418] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.106891] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=20] ====== tenant freeze timer task ====== [2025-05-15 09:40:40.106939] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:687) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=28) [2025-05-15 09:40:40.106963] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:593) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=21] TxData Memory Statistic : (Tenant Total Memory(MB)=5120, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=28, Freeze TxData Trigger Memory(MB)=102, Total TxDataTable Hold Memory(MB)=36, Total TxDataTable Memory Limit(MB)=1024) [2025-05-15 09:40:40.106988] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.107013] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040106983) [2025-05-15 09:40:40.107028] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302040006978, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.107056] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12342355096, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.107073] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.107095] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.107111] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.107121] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.107137] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12342355096, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.107150] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.107165] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.107177] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.107188] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.107201] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.107211] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.107218] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040107041) [2025-05-15 09:40:40.115830] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:40.115847] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:40.115856] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=7] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:40.115872] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:40.116528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.118868] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.118887] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.118906] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.118926] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.118941] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.118954] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.118967] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.118980] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.118993] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302040118847, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.119007] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.119014] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.119079] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.119090] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.119100] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.119112] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.120417] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:40.124048] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.124070] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.124086] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.126643] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.128649] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.128687] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040128643) [2025-05-15 09:40:40.128702] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302039928582, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.128725] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12342400647, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.128721] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.128740] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040128717) [2025-05-15 09:40:40.128749] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302040028712, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.128742] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.128745] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221553179, support_ssl=false) [2025-05-15 09:40:40.128765] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:40.128775] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12342228154, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.128777] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.128788] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.128790] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.128800] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.128809] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.128805] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:40.128811] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553179, sess.client_addr_="172.22.0.1:39904") [2025-05-15 09:40:40.128815] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040128712) [2025-05-15 09:40:40.128817] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.128826] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.128826] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553179, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.128841] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.128852] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.128859] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040128764) [2025-05-15 09:40:40.128861] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:40.128850] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221553180, support_ssl=false) [2025-05-15 09:40:40.128882] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] temporary file module use no blocks [2025-05-15 09:40:40.128899] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:40.128917] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:40.128938] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=43] sm conn init succ(conn.sessid_=3221553180, sess.client_addr_="172.22.0.1:39908") [2025-05-15 09:40:40.128930] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=13] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11803019281, flush_io_finished_ret_:0, flush_io_finished_round_:1514, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:40.128962] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553180, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.129011] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.129044] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.129064] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.129064] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=31][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.129083] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.129093] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.129105] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.129111] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.129126] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.129142] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553179, proxy_sessid=0) [2025-05-15 09:40:40.129142] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.129169] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=9][errcode=0] server close connection(sessid=3221553179, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.129181] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553179, proxy_sessid=0) [2025-05-15 09:40:40.129182] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=38][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.129189] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.129194] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.129202] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.129197] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909E-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553179, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.129229] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.129249] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553180, proxy_sessid=0) [2025-05-15 09:40:40.129248] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553179, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302040128482, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.129271] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553179, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.129280] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553179, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302040128482, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.129284] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=16][errcode=0] server close connection(sessid=3221553180, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.129298] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553180, proxy_sessid=0) [2025-05-15 09:40:40.129298] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553179) [2025-05-15 09:40:40.129305] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.129314] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908D-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553180, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.129315] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553179, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.129369] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553180, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302040128482, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.129399] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221553180, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.129409] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553180, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302040128482, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.129435] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] mark session id unused(conn.sessid_=3221553180) [2025-05-15 09:40:40.129452] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553181, support_ssl=false) [2025-05-15 09:40:40.129453] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553180, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.129500] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553181, sess.client_addr_="172.22.0.1:39920") [2025-05-15 09:40:40.129518] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553181, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.129721] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.129751] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.129766] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.129780] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.129795] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.129802] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.129817] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.129831] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553181, proxy_sessid=0) [2025-05-15 09:40:40.129850] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=8][errcode=0] server close connection(sessid=3221553181, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.129862] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553181, proxy_sessid=0) [2025-05-15 09:40:40.129869] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:40.129880] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CE-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553181, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.129923] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553181, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040129541, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.129944] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553181, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.129955] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553181, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040129541, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.129972] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553181) [2025-05-15 09:40:40.129985] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553181, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.130147] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553182, support_ssl=false) [2025-05-15 09:40:40.130239] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=43] sm conn init succ(conn.sessid_=3221553182, sess.client_addr_="172.22.0.1:39936") [2025-05-15 09:40:40.130257] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553182, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.130431] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.130467] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.130486] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.130498] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553183, support_ssl=false) [2025-05-15 09:40:40.130511] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.130518] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.130524] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.130527] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553183, sess.client_addr_="172.22.0.1:39944") [2025-05-15 09:40:40.130537] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.130548] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553182, proxy_sessid=0) [2025-05-15 09:40:40.130543] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553183, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.130570] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=12][errcode=0] server close connection(sessid=3221553182, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.130580] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553182, proxy_sessid=0) [2025-05-15 09:40:40.130594] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:40.130600] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC909F-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553182, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.130654] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553182, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302040129541, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.130681] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553182, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.130689] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553182, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302040129541, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.130702] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] mark session id unused(conn.sessid_=3221553182) [2025-05-15 09:40:40.130724] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553182, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.130766] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.130792] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.130811] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.130831] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.130839] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.130852] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.130865] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.130880] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553183, proxy_sessid=0) [2025-05-15 09:40:40.130899] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=8][errcode=0] server close connection(sessid=3221553183, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.130913] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553183, proxy_sessid=0) [2025-05-15 09:40:40.130921] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.130930] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553184, support_ssl=false) [2025-05-15 09:40:40.130934] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908E-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553183, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.130967] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553184, sess.client_addr_="172.22.0.1:39950") [2025-05-15 09:40:40.130980] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553183, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302040130601, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.130983] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553184, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.131005] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553183, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.131014] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553183, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302040130601, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.131030] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553183) [2025-05-15 09:40:40.131046] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553183, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.131229] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.131263] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.131281] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.131297] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.131305] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.131311] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.131323] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.131323] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553185, support_ssl=false) [2025-05-15 09:40:40.131338] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553184, proxy_sessid=0) [2025-05-15 09:40:40.131356] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=8][errcode=0] server close connection(sessid=3221553184, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.131369] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553185, sess.client_addr_="172.22.0.1:39952") [2025-05-15 09:40:40.131369] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221553186, support_ssl=false) [2025-05-15 09:40:40.131368] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553184, proxy_sessid=0) [2025-05-15 09:40:40.131388] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=21][errcode=-4016] session is null [2025-05-15 09:40:40.131387] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553185, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.131395] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90CF-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553184, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.131408] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221553186, sess.client_addr_="172.22.0.1:39968") [2025-05-15 09:40:40.131431] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553186, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.131464] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553184, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040130601, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.131497] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553184, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.131515] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553184, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040130601, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.131537] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221553184) [2025-05-15 09:40:40.131550] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553184, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.131790] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.131818] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.131819] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.131836] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.131852] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.131849] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.131860] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.131866] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=14][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.131871] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.131883] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.131887] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.131891] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.131899] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.131897] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553186, proxy_sessid=0) [2025-05-15 09:40:40.131912] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.131918] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=11][errcode=0] server close connection(sessid=3221553186, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.131929] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553185, proxy_sessid=0) [2025-05-15 09:40:40.131931] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553186, proxy_sessid=0) [2025-05-15 09:40:40.131940] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:40.131948] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=9][errcode=0] server close connection(sessid=3221553185, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.131948] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A0-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553186, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.131962] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553185, proxy_sessid=0) [2025-05-15 09:40:40.131970] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.131961] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553187, support_ssl=false) [2025-05-15 09:40:40.131976] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC908F-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553185, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.132006] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553186, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132027] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553185, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132036] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221553186, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.132040] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553187, sess.client_addr_="172.22.0.1:39972") [2025-05-15 09:40:40.132053] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553185, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.132054] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553186, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.132062] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553185, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.132070] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553186) [2025-05-15 09:40:40.132059] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553187, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132083] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553185) [2025-05-15 09:40:40.132087] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553186, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.132095] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553188, support_ssl=false) [2025-05-15 09:40:40.132096] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553185, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.132132] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553189, support_ssl=false) [2025-05-15 09:40:40.132127] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553188, sess.client_addr_="172.22.0.1:39974") [2025-05-15 09:40:40.132156] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553188, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132172] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553189, sess.client_addr_="172.22.0.1:39976") [2025-05-15 09:40:40.132190] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553189, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132254] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.132293] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.132308] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.132326] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.132323] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.132334] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.132332] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.132341] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.132350] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.132355] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.132361] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.132368] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.132380] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553187, proxy_sessid=0) [2025-05-15 09:40:40.132383] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.132384] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.132397] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.132407] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.132408] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.132412] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=16][errcode=0] server close connection(sessid=3221553187, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.132427] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553187, proxy_sessid=0) [2025-05-15 09:40:40.132432] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=17] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.132428] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.132438] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:40.132449] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553189, proxy_sessid=0) [2025-05-15 09:40:40.132451] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=23][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.132453] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D0-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553187, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.132477] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=15][errcode=0] server close connection(sessid=3221553189, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.132502] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=24][errcode=-4018] get session fail(ret=-4018, sessid=3221553189, proxy_sessid=0) [2025-05-15 09:40:40.132517] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553187, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132523] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=21][errcode=-4016] session is null [2025-05-15 09:40:40.132525] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=26] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.132545] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221553187, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.132556] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553187, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.132552] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=25][errcode=-4018] get session fail(ret=-4018, sessid=3221553188, proxy_sessid=0) [2025-05-15 09:40:40.132543] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A1-0-0] [lt=19] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553189, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.132577] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553187) [2025-05-15 09:40:40.132592] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=23][errcode=0] server close connection(sessid=3221553188, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.132593] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553187, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.132612] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=19][errcode=-4018] get session fail(ret=-4018, sessid=3221553188, proxy_sessid=0) [2025-05-15 09:40:40.132614] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553189, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132628] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=15][errcode=-4016] session is null [2025-05-15 09:40:40.132644] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] kill and revert session(conn.sessid_=3221553189, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.132643] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9090-0-0] [lt=14] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553188, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.132661] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553189, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.132681] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553189) [2025-05-15 09:40:40.132694] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553189, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.132698] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553188, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.132719] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553188, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.132759] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553188, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302040131662, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.132777] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553188) [2025-05-15 09:40:40.132789] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553188, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.136743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.144163] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:40.146862] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.156970] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.167078] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.174108] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.174254] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=22] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.174327] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.174388] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=18] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.177187] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.187296] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.191596] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.191622] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4080106, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.191640] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=4080106, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.191655] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.191667] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=4080106, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.191681] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.191816] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=133][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.191832] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.191848] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.191860] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.191872] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.191890] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040191888, replica_locations:[]}) [2025-05-15 09:40:40.191908] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.191935] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.191947] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:40.191978] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.192014] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:40.192033] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11804082510, cache_obj->added_lc()=false, cache_obj->get_object_id()=15826, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.193367] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.193395] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.193412] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.193426] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.193435] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.193448] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.193463] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.193476] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.193498] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.193509] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.193523] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.193537] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040193536, replica_locations:[]}) [2025-05-15 09:40:40.193603] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=4078079, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=304, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.197406] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.198370] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=19] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:40.198400] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:40.198420] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=19][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:40.198466] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=45][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:40.198477] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:40.198498] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=21][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:40.198511] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:40.198523] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:40.198534] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:40.198546] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=12][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:40.198558] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:40.198569] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:40.198580] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:40.198590] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:40.198612] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:40.198623] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.198637] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.198649] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:40.198661] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:40.198676] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:40.198688] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:40.198708] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:40.198731] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:40.198741] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:40.198747] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:40.198767] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:40.198781] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_LSMetaCh][T1][YB427F000001-000635291D5C6445-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.198795] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=13][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:40.198807] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.198819] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:40.198830] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:40.198841] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302040198116, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.198858] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:40.198869] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.198959] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=12][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:40.198978] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=18][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:40.198991] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=12][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:40.199002] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:40.199015] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:40.199029] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:40.199040] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4461][T1001_TimerWK1_LSMetaCh][T1001][YB427F000001-000635291D5C6445-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:40.206105] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.207070] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12342455583, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.207097] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.207123] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.207143] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.207155] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.207173] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12342455583, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.207186] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.207201] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.207213] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.207221] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.207235] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.207245] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.207256] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040207043) [2025-05-15 09:40:40.207302] INFO [COMMON] generate_mod_stat_task (memory_dump.cpp:212) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] task info(*task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-15 09:40:40.207330] INFO [COMMON] handle (memory_dump.cpp:531) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=26] handle dump task(task={type_:2, dump_all_:false, p_context_:null, slot_idx_:0, dump_tenant_ctx_:false, tenant_id_:0, ctx_id_:0, p_chunk_:null}) [2025-05-15 09:40:40.207514] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.207701] INFO [COMMON] update_check_range (ob_sql_mem_leak_checker.cpp:62) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=13] update_check_range(min_check_version=150, max_check_version=151, global_version=152) [2025-05-15 09:40:40.217627] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.219081] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.219116] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=33][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.219141] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.219161] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.219176] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.219214] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=37][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.219230] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.219240] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.219250] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302040219037, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.219266] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.219280] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.219367] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.219380] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.219394] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.219409] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.220995] INFO handle (memory_dump.cpp:658) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=14] statistics: tenant_cnt: 6, max_chunk_cnt: 524288 tenant_id ctx_id chunk_cnt label_cnt segv_cnt 1 0 116 220 0 1 2 57 1 0 1 4 1 1 0 1 5 1 3 0 1 7 1 2 0 1 8 43 1 0 1 12 3 2 0 1 13 2 1 0 1 18 11 3 0 1 20 1 1 0 500 0 57 222 0 500 7 9 4 0 500 8 49 1 0 500 9 2 1 0 500 10 2 1 0 500 17 6 3 0 500 22 3 50 0 500 23 23 10 0 508 0 2 7 0 508 8 15 1 0 509 0 2 7 0 509 8 9 1 0 1001 0 74 205 0 1001 2 35 1 0 1001 4 1 1 0 1001 5 1 1 0 1001 8 29 1 0 1001 12 3 3 0 1001 13 1 1 0 1001 18 17 3 0 1001 20 2 1 0 1002 0 82 206 0 1002 2 90 1 0 1002 4 1 1 0 1002 5 1 1 0 1002 8 63 1 0 1002 12 2 2 0 1002 13 1 1 0 1002 18 5 3 0 cost_time: 13349 [2025-05-15 09:40:40.221052] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=28] [MEMORY] tenant: 1, limit: 5368709120 hold: 850857984 cache_hold: 140509184 cache_used: 140509184 cache_item_count: 67 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 410435584 limit= 9223372036854775807 [MEMORY] ctx_id= MEMSTORE_CTX_ID hold_bytes= 119537664 limit= 9223372036854775807 [MEMORY] ctx_id= TRANS_CTX_MGR_ID hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= PLAN_CACHE_CTX_ID hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 90177536 limit= 9223372036854775807 [MEMORY] ctx_id= META_OBJ_CTX_ID hold_bytes= 6291456 limit= 1073741820 [MEMORY] ctx_id= TX_CALLBACK_CTX_ID hold_bytes= 4194304 limit= 9223372036854775807 [MEMORY] ctx_id= TX_DATA_TABLE hold_bytes= 37769216 limit= 9223372036854775807 [MEMORY] ctx_id= MDS_DATA_ID hold_bytes= 2097152 limit= 9223372036854775807 [MEMORY] ctx_id= MERGE_RESERVE_CTX_ID hold_bytes= 33554432 limit= 9223372036854775807 [2025-05-15 09:40:40.221339] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=15] [MEMORY] tenant_id= 1 ctx_id= DEFAULT_CTX_ID hold= 410435584 used= 397620688 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 82821120 used= 82569792 count= 62 avg_used= 1331770 block_cnt= 62 chunk_cnt= 44 mod=MysqlRequesReco [MEMORY] hold= 41963520 used= 41943040 count= 1 avg_used= 41943040 block_cnt= 1 chunk_cnt= 1 mod=LogGroupBuffer [MEMORY] hold= 31477760 used= 31458304 count= 1 avg_used= 31458304 block_cnt= 1 chunk_cnt= 1 mod=ASHListBuffer [MEMORY] hold= 31428608 used= 31368256 count= 7 avg_used= 4481179 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 28008448 used= 27926528 count= 4 avg_used= 6981632 block_cnt= 4 chunk_cnt= 4 mod=ServerObjecPool [MEMORY] hold= 16232912 used= 16067217 count= 111 avg_used= 144749 block_cnt= 38 chunk_cnt= 8 mod=OmtTenant [MEMORY] hold= 9859072 used= 9831040 count= 2 avg_used= 4915520 block_cnt= 2 chunk_cnt= 2 mod=HashBuckDmId [MEMORY] hold= 9510912 used= 9437472 count= 9 avg_used= 1048608 block_cnt= 9 chunk_cnt= 9 mod=DiagnosticInfos [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDmChe [MEMORY] hold= 8777728 used= 8760064 count= 2 avg_used= 4380032 block_cnt= 2 chunk_cnt= 2 mod=FreeTbltStream [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=RCSrv [MEMORY] hold= 8540160 used= 8519680 count= 1 avg_used= 8519680 block_cnt= 1 chunk_cnt= 1 mod=ArcFetchQueue [MEMORY] hold= 5386240 used= 5368320 count= 1 avg_used= 5368320 block_cnt= 1 chunk_cnt= 1 mod=RespTimeColl [MEMORY] hold= 5308416 used= 4484160 count= 162 avg_used= 27680 block_cnt= 162 chunk_cnt= 21 mod=DiagnosticInfo [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMonMap [MEMORY] hold= 4235264 used= 4194304 count= 2 avg_used= 2097152 block_cnt= 2 chunk_cnt= 2 mod=SHARE_BLOCK [MEMORY] hold= 4214784 used= 4194336 count= 1 avg_used= 4194336 block_cnt= 1 chunk_cnt= 1 mod=BloomFilter [MEMORY] hold= 4161536 used= 4159488 count= 2 avg_used= 2079744 block_cnt= 2 chunk_cnt= 2 mod=CACHE_MAP_NODE [MEMORY] hold= 3879456 used= 3842280 count= 89 avg_used= 43171 block_cnt= 89 chunk_cnt= 4 mod=TabletMap [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMem [MEMORY] hold= 3865600 used= 3705600 count= 800 avg_used= 4632 block_cnt= 800 chunk_cnt= 12 mod=CkptDgnMemCU [MEMORY] hold= 3772416 used= 3751944 count= 1 avg_used= 3751944 block_cnt= 1 chunk_cnt= 1 mod=IteratorStorage [MEMORY] hold= 3723264 used= 3698704 count= 2 avg_used= 1849352 block_cnt= 2 chunk_cnt= 2 mod=LogDIOAligned [MEMORY] hold= 3162112 used= 3145808 count= 2 avg_used= 1572904 block_cnt= 2 chunk_cnt= 2 mod=TenantDLMgr [MEMORY] hold= 3145728 used= 2228224 count= 128 avg_used= 17408 block_cnt= 128 chunk_cnt= 6 mod=SqlDtlQueue [MEMORY] hold= 3047424 used= 3013468 count= 10 avg_used= 301346 block_cnt= 10 chunk_cnt= 3 mod=LDIOSetup [MEMORY] hold= 3002368 used= 2982720 count= 1 avg_used= 2982720 block_cnt= 1 chunk_cnt= 1 mod=RpcStatInfo [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=HashBuckDTLINT [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=MediumTabletMap [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=FastFrezCkr [MEMORY] hold= 2375680 used= 2359536 count= 2 avg_used= 1179768 block_cnt= 2 chunk_cnt= 2 mod=HashBuckLCSta [MEMORY] hold= 2248704 used= 2228224 count= 1 avg_used= 2228224 block_cnt= 1 chunk_cnt= 1 mod=LogIOCb [MEMORY] hold= 2198768 used= 416700 count= 25277 avg_used= 16 block_cnt= 271 chunk_cnt= 3 mod=Number [MEMORY] hold= 2150400 used= 2131072 count= 1 avg_used= 2131072 block_cnt= 1 chunk_cnt= 1 mod=TransService [MEMORY] hold= 2097152 used= 2087424 count= 2 avg_used= 1043712 block_cnt= 2 chunk_cnt= 2 mod=DiagnosticSum [MEMORY] hold= 1843200 used= 1770696 count= 9 avg_used= 196744 block_cnt= 9 chunk_cnt= 5 mod=DagMap [MEMORY] hold= 1584544 used= 1575992 count= 3 avg_used= 525330 block_cnt= 2 chunk_cnt= 2 mod=PartitTableTask [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=XATimeWheel [MEMORY] hold= 1294336 used= 1280384 count= 2 avg_used= 640192 block_cnt= 2 chunk_cnt= 1 mod=TransTimeWheel [MEMORY] hold= 1269568 used= 1263904 count= 6 avg_used= 210650 block_cnt= 6 chunk_cnt= 3 mod=PoolFreeList [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=HashBuckPlanCac [MEMORY] hold= 1187840 used= 1179768 count= 1 avg_used= 1179768 block_cnt= 1 chunk_cnt= 1 mod=RewriteRuleMap [MEMORY] hold= 958464 used= 950272 count= 1 avg_used= 950272 block_cnt= 1 chunk_cnt= 1 mod=IOWorkerLQ [MEMORY] hold= 933888 used= 931072 count= 1 avg_used= 931072 block_cnt= 1 chunk_cnt= 1 mod=ArcSenderQueue [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSMap [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlPlanMon [MEMORY] hold= 802816 used= 800000 count= 1 avg_used= 800000 block_cnt= 1 chunk_cnt= 1 mod=SqlFltSpanRec [MEMORY] hold= 786688 used= 524352 count= 33 avg_used= 15889 block_cnt= 33 chunk_cnt= 4 mod=LogAlloc [MEMORY] hold= 739648 used= 722176 count= 91 avg_used= 7936 block_cnt= 91 chunk_cnt= 8 mod=TltMemtablMgr [MEMORY] hold= 737280 used= 655360 count= 10 avg_used= 65536 block_cnt= 10 chunk_cnt= 6 mod=[T]char [MEMORY] hold= 663552 used= 655360 count= 1 avg_used= 655360 block_cnt= 1 chunk_cnt= 1 mod=FetchLog [MEMORY] hold= 663552 used= 659200 count= 1 avg_used= 659200 block_cnt= 1 chunk_cnt= 1 mod=MulLevelQueue [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DupTbLease [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=FrzTrigger [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTF [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=CoordTR [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=ElectTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=OBJLockGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MultiVersionGC [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=MdsT [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=DetectorTimer [MEMORY] hold= 598016 used= 590232 count= 1 avg_used= 590232 block_cnt= 1 chunk_cnt= 1 mod=DagNetIdMap [MEMORY] hold= 491520 used= 468576 count= 3 avg_used= 156192 block_cnt= 3 chunk_cnt= 2 mod=SQLSessionInfo [MEMORY] hold= 475136 used= 458816 count= 2 avg_used= 229408 block_cnt= 2 chunk_cnt= 2 mod=ts_token [MEMORY] hold= 417664 used= 412408 count= 4 avg_used= 103102 block_cnt= 4 chunk_cnt= 1 mod=ResultSet [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ReplaySrv [MEMORY] hold= 409600 used= 401408 count= 1 avg_used= 401408 block_cnt= 1 chunk_cnt= 1 mod=ApplySrv [MEMORY] hold= 401408 used= 393504 count= 1 avg_used= 393504 block_cnt= 1 chunk_cnt= 1 mod=ClogGe [MEMORY] hold= 401408 used= 393248 count= 1 avg_used= 393248 block_cnt= 1 chunk_cnt= 1 mod=T3MBucket [MEMORY] hold= 401408 used= 393248 count= 1 avg_used= 393248 block_cnt= 1 chunk_cnt= 1 mod=ResourMapLock [MEMORY] hold= 385024 used= 375520 count= 2 avg_used= 187760 block_cnt= 2 chunk_cnt= 2 mod=bf_queue [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=DmlStatHashMap [MEMORY] hold= 303104 used= 294936 count= 1 avg_used= 294936 block_cnt= 1 chunk_cnt= 1 mod=ColUsagHashMap [MEMORY] hold= 286720 used= 280000 count= 1 avg_used= 280000 block_cnt= 1 chunk_cnt= 1 mod=MetaMemMgr [MEMORY] hold= 246880 used= 3318 count= 3061 avg_used= 1 block_cnt= 31 chunk_cnt= 1 mod=CharsetUtil [MEMORY] hold= 212992 used= 211072 count= 1 avg_used= 211072 block_cnt= 1 chunk_cnt= 1 mod=TimerWK [MEMORY] hold= 204800 used= 196744 count= 1 avg_used= 196744 block_cnt= 1 chunk_cnt= 1 mod=DagNetMap [MEMORY] hold= 204800 used= 196616 count= 1 avg_used= 196616 block_cnt= 1 chunk_cnt= 1 mod=TenDLBucket [MEMORY] hold= 196608 used= 195840 count= 3 avg_used= 65280 block_cnt= 3 chunk_cnt= 1 mod=MemTblObj [MEMORY] hold= 180224 used= 105600 count= 11 avg_used= 9600 block_cnt= 11 chunk_cnt= 6 mod=[T]ObDSActionAr [MEMORY] hold= 147456 used= 139264 count= 1 avg_used= 139264 block_cnt= 1 chunk_cnt= 1 mod=RFLTaskQueue [MEMORY] hold= 147328 used= 141880 count= 3 avg_used= 47293 block_cnt= 3 chunk_cnt= 1 mod=CommSysVarFac [MEMORY] hold= 122880 used= 110608 count= 2 avg_used= 55304 block_cnt= 2 chunk_cnt= 1 mod=TabletIDSetBkt [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=DbmsSched_Job [MEMORY] hold= 114688 used= 111096 count= 1 avg_used= 111096 block_cnt= 1 chunk_cnt= 1 mod=AsyncFreezeTabl [MEMORY] hold= 114688 used= 106496 count= 1 avg_used= 106496 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlMgr [MEMORY] hold= 104000 used= 101504 count= 13 avg_used= 7808 block_cnt= 13 chunk_cnt= 2 mod=TabletIDSetNode [MEMORY] hold= 98304 used= 94752 count= 1 avg_used= 94752 block_cnt= 1 chunk_cnt= 1 mod=LSSvr [MEMORY] hold= 97472 used= 84160 count= 3 avg_used= 28053 block_cnt= 3 chunk_cnt= 3 mod=DI_CONTAINER [MEMORY] hold= 82064 used= 22016 count= 299 avg_used= 73 block_cnt= 63 chunk_cnt= 18 mod=Coro [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_cancel_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_run_set [MEMORY] hold= 81824 used= 73472 count= 2 avg_used= 36736 block_cnt= 2 chunk_cnt= 2 mod=ts_queue [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSharedQueueT [MEMORY] hold= 73728 used= 72736 count= 1 avg_used= 72736 block_cnt= 1 chunk_cnt= 1 mod=LogSubmitLogQue [MEMORY] hold= 73728 used= 66176 count= 1 avg_used= 66176 block_cnt= 1 chunk_cnt= 1 mod=Rpc [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=LockMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxDataMemObj [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=DEVICE_MANAGER [MEMORY] hold= 65536 used= 65280 count= 1 avg_used= 65280 block_cnt= 1 chunk_cnt= 1 mod=TxCtxMemObj [MEMORY] hold= 65408 used= 63488 count= 4 avg_used= 15872 block_cnt= 4 chunk_cnt= 2 mod=IOUsageInfo [MEMORY] hold= 57344 used= 51328 count= 1 avg_used= 51328 block_cnt= 1 chunk_cnt= 1 mod=palf_env [MEMORY] hold= 49152 used= 42496 count= 1 avg_used= 42496 block_cnt= 1 chunk_cnt= 1 mod=IO_MGR [MEMORY] hold= 48960 used= 44840 count= 2 avg_used= 22420 block_cnt= 2 chunk_cnt= 2 mod=DynamicFactor [MEMORY] hold= 47360 used= 45440 count= 10 avg_used= 4544 block_cnt= 10 chunk_cnt= 6 mod=[T]ObTraceEvent [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CliTaskMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=DagWarnHisBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=SuspectInfoBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletPtrMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=Autoincrement [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_CTBriefMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TLD_TblCtxMap [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HTblRkMapBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=TableAQueryBkt [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HashBucCliInFo [MEMORY] hold= 40960 used= 37032 count= 1 avg_used= 37032 block_cnt= 1 chunk_cnt= 1 mod=HB_SERVICE [MEMORY] hold= 40640 used= 39680 count= 5 avg_used= 7936 block_cnt= 5 chunk_cnt= 1 mod=ObDMMDL [MEMORY] hold= 37968 used= 4056 count= 169 avg_used= 24 block_cnt= 46 chunk_cnt= 16 mod=[T]MemoryContex [MEMORY] hold= 32704 used= 25191 count= 2 avg_used= 12595 block_cnt= 2 chunk_cnt= 2 mod=CGReadInfoMgr [MEMORY] hold= 25600 used= 17920 count= 40 avg_used= 448 block_cnt= 16 chunk_cnt= 3 mod=OMT_Worker [MEMORY] hold= 24576 used= 16392 count= 1 avg_used= 16392 block_cnt= 1 chunk_cnt= 1 mod=SlogWriteBuffer [MEMORY] hold= 24384 used= 23808 count= 3 avg_used= 7936 block_cnt= 3 chunk_cnt= 1 mod=SqlSession [MEMORY] hold= 17024 used= 8448 count= 3 avg_used= 2816 block_cnt= 2 chunk_cnt= 2 mod=BaseLogWriter [MEMORY] hold= 16384 used= 14016 count= 1 avg_used= 14016 block_cnt= 1 chunk_cnt= 1 mod=LogReplayStatus [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=PlGdbHelper [MEMORY] hold= 16384 used= 11584 count= 1 avg_used= 11584 block_cnt= 1 chunk_cnt= 1 mod=LogApplyStatus [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MaBlkLoadMap [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=DiaTabletMap [MEMORY] hold= 16384 used= 9792 count= 1 avg_used= 9792 block_cnt= 1 chunk_cnt= 1 mod=TSQLSessionMgr [MEMORY] hold= 16384 used= 13896 count= 1 avg_used= 13896 block_cnt= 1 chunk_cnt= 1 mod=task_cache [MEMORY] hold= 16384 used= 10248 count= 1 avg_used= 10248 block_cnt= 1 chunk_cnt= 1 mod=TabletSvrBucket [MEMORY] hold= 16384 used= 8200 count= 1 avg_used= 8200 block_cnt= 1 chunk_cnt= 1 mod=SlogNopLog [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TabletStats [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=backupTaskSched [MEMORY] hold= 16384 used= 8448 count= 1 avg_used= 8448 block_cnt= 1 chunk_cnt= 1 mod=PalfEnv [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=TLD_AssignedMgr [MEMORY] hold= 16384 used= 12344 count= 1 avg_used= 12344 block_cnt= 1 chunk_cnt= 1 mod=MedCheckSet [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapDi [MEMORY] hold= 12864 used= 12288 count= 3 avg_used= 4096 block_cnt= 3 chunk_cnt= 3 mod=LinearHashMapCn [MEMORY] hold= 9456 used= 9064 count= 2 avg_used= 4532 block_cnt= 2 chunk_cnt= 1 mod=GROUP_INDEX_MAP [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=RawPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CtxBlockPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ConstPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DictPl [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ShareBlksMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=CPUUSage [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=leakChecker [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=APPLY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=REPLAY_STATUS [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=HTableLockMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MatchLsnMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=IORunners [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsTableMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=DASIDCache [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LCLSender [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=LockWaitMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=ls_id [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileBlkMgr [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TmpFileMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=PlanVaIdx [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiVarMap [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=MdsDebugMap [MEMORY] hold= 8000 used= 7808 count= 1 avg_used= 7808 block_cnt= 1 chunk_cnt= 1 mod=HashNodeLCSta [MEMORY] hold= 7296 used= 6912 count= 2 avg_used= 3456 block_cnt= 2 chunk_cnt= 2 mod=CACHE_INST [MEMORY] hold= 5056 used= 4656 count= 2 avg_used= 2328 block_cnt= 2 chunk_cnt= 2 mod=MViewService [MEMORY] hold= 4848 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=PxPoolBkt [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResRuleIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ResGrpIdMap [MEMORY] hold= 4832 used= 4632 count= 1 avg_used= 4632 block_cnt= 1 chunk_cnt= 1 mod=ExTabletCntMap [MEMORY] hold= 4288 used= 4096 count= 1 avg_used= 4096 block_cnt= 1 chunk_cnt= 1 mod=MacroFile [MEMORY] hold= 4160 used= 3520 count= 4 avg_used= 880 block_cnt= 3 chunk_cnt= 3 mod=PoolArenaArray [MEMORY] hold= 3328 used= 1792 count= 8 avg_used= 224 block_cnt= 1 chunk_cnt= 1 mod=LogIOTask [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=PendingFreeMap [MEMORY] hold= 3312 used= 3112 count= 1 avg_used= 3112 block_cnt= 1 chunk_cnt= 1 mod=TFileFLCtx [MEMORY] hold= 3200 used= 3008 count= 1 avg_used= 3008 block_cnt= 1 chunk_cnt= 1 mod=SqlDtlDfc [MEMORY] hold= 2880 used= 2400 count= 3 avg_used= 800 block_cnt= 3 chunk_cnt= 2 mod=DICollector [MEMORY] hold= 2592 used= 2016 count= 3 avg_used= 672 block_cnt= 2 chunk_cnt= 2 mod=[T]ObWarningBuf [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=MedCheckMap [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=VecIdxLSMgr [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=DASMemHashBuck [MEMORY] hold= 2528 used= 2328 count= 1 avg_used= 2328 block_cnt= 1 chunk_cnt= 1 mod=IRMMemHashBuck [MEMORY] hold= 2048 used= 1856 count= 1 avg_used= 1856 block_cnt= 1 chunk_cnt= 1 mod=LogIOWS [MEMORY] hold= 2016 used= 1824 count= 1 avg_used= 1824 block_cnt= 1 chunk_cnt= 1 mod=PxResMgr [MEMORY] hold= 1920 used= 1728 count= 1 avg_used= 1728 block_cnt= 1 chunk_cnt= 1 mod=timer_service [MEMORY] hold= 1808 used= 1424 count= 2 avg_used= 712 block_cnt= 2 chunk_cnt= 2 mod=DetectManager [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TXBFRetryId [MEMORY] hold= 1744 used= 1544 count= 1 avg_used= 1544 block_cnt= 1 chunk_cnt= 1 mod=TabStatMgrLock [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=HashBucApiGroup [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=GCMemtableMap [MEMORY] hold= 1472 used= 1272 count= 1 avg_used= 1272 block_cnt= 1 chunk_cnt= 1 mod=DUP_LS_SET [MEMORY] hold= 1280 used= 1088 count= 1 avg_used= 1088 block_cnt= 1 chunk_cnt= 1 mod=ModuleInitCtx [MEMORY] hold= 1248 used= 1056 count= 1 avg_used= 1056 block_cnt= 1 chunk_cnt= 1 mod=LOG_HASH_MAP [MEMORY] hold= 1232 used= 1032 count= 1 avg_used= 1032 block_cnt= 1 chunk_cnt= 1 mod=BFLoadTask [MEMORY] hold= 1200 used= 1000 count= 1 avg_used= 1000 block_cnt= 1 chunk_cnt= 1 mod=FlyTabletMapLk [MEMORY] hold= 1120 used= 120 count= 5 avg_used= 24 block_cnt= 1 chunk_cnt= 1 mod=FreezeTask [MEMORY] hold= 976 used= 776 count= 1 avg_used= 776 block_cnt= 1 chunk_cnt= 1 mod=DepTabletSet [MEMORY] hold= 896 used= 696 count= 1 avg_used= 696 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfo [MEMORY] hold= 752 used= 552 count= 1 avg_used= 552 block_cnt= 1 chunk_cnt= 1 mod=LSFreeze [MEMORY] hold= 704 used= 512 count= 1 avg_used= 512 block_cnt= 1 chunk_cnt= 1 mod=SqlSessiQuerSql [MEMORY] hold= 672 used= 472 count= 1 avg_used= 472 block_cnt= 1 chunk_cnt= 1 mod=Election [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=PackStateMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=SequenceIdMap [MEMORY] hold= 624 used= 424 count= 1 avg_used= 424 block_cnt= 1 chunk_cnt= 1 mod=ContextsMap [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=HAScheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=Scheduler [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=VectorIndexVsag [MEMORY] hold= 576 used= 384 count= 1 avg_used= 384 block_cnt= 1 chunk_cnt= 1 mod=MSTXCTX [MEMORY] hold= 544 used= 144 count= 2 avg_used= 72 block_cnt= 1 chunk_cnt= 1 mod=TntSrvObjPool [MEMORY] hold= 448 used= 64 count= 2 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfFSCbNode [MEMORY] hold= 416 used= 2 count= 2 avg_used= 1 block_cnt= 2 chunk_cnt= 2 mod=ObLogEXTHandler [MEMORY] hold= 400 used= 200 count= 1 avg_used= 200 block_cnt= 1 chunk_cnt= 1 mod=ExTabletMapLk [MEMORY] hold= 336 used= 144 count= 1 avg_used= 144 block_cnt= 1 chunk_cnt= 1 mod=ObTxLSLogCb [MEMORY] hold= 304 used= 112 count= 1 avg_used= 112 block_cnt= 1 chunk_cnt= 1 mod=Coordinator [MEMORY] hold= 304 used= 104 count= 1 avg_used= 104 block_cnt= 1 chunk_cnt= 1 mod=RedoLogBuckLock [MEMORY] hold= 256 used= 56 count= 1 avg_used= 56 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 240 used= 41 count= 1 avg_used= 41 block_cnt= 1 chunk_cnt= 1 mod=KeepAliveBuf [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PalfRCCbNode [MEMORY] hold= 224 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=RebuildCbNode [MEMORY] hold= 208 used= 16 count= 1 avg_used= 16 block_cnt= 1 chunk_cnt= 1 mod=logservice [MEMORY] hold= 397620688 used= 391186209 count= 31479 avg_used= 12426 mod=SUMMARY [2025-05-15 09:40:40.221373] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=24] [MEMORY] tenant_id= 1 ctx_id= MEMSTORE_CTX_ID hold= 119537664 used= 118603776 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 118603776 used= 118549512 count= 57 avg_used= 2079816 block_cnt= 57 chunk_cnt= 57 mod=Memstore [MEMORY] hold= 118603776 used= 118549512 count= 57 avg_used= 2079816 mod=SUMMARY [2025-05-15 09:40:40.221388] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=7] [MEMORY] tenant_id= 1 ctx_id= TRANS_CTX_MGR_ID hold= 2097152 used= 466944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 466944 used= 460416 count= 1 avg_used= 460416 block_cnt= 1 chunk_cnt= 1 mod=PartTranCtxMgr [MEMORY] hold= 466944 used= 460416 count= 1 avg_used= 460416 mod=SUMMARY [2025-05-15 09:40:40.221404] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=8] [MEMORY] tenant_id= 1 ctx_id= PLAN_CACHE_CTX_ID hold= 2097152 used= 36928 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 32768 used= 25864 count= 1 avg_used= 25864 block_cnt= 1 chunk_cnt= 1 mod=SqlPhyPlan [MEMORY] hold= 2176 used= 1984 count= 1 avg_used= 1984 block_cnt= 1 chunk_cnt= 1 mod=CreateContext [MEMORY] hold= 1984 used= 1600 count= 2 avg_used= 800 block_cnt= 1 chunk_cnt= 1 mod=PlanCache [MEMORY] hold= 36928 used= 29448 count= 4 avg_used= 7362 mod=SUMMARY [2025-05-15 09:40:40.221425] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= GLIBC hold= 2097152 used= 80496 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 80400 used= 58751 count= 259 avg_used= 226 block_cnt= 20 chunk_cnt= 1 mod=PlJit [MEMORY] hold= 96 used= 32 count= 1 avg_used= 32 block_cnt= 1 chunk_cnt= 1 mod=PlCodeGen [MEMORY] hold= 80496 used= 58783 count= 260 avg_used= 226 mod=SUMMARY [2025-05-15 09:40:40.221441] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= CO_STACK hold= 90177536 used= 88768512 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 88768512 used= 88603392 count= 172 avg_used= 515136 block_cnt= 172 chunk_cnt= 43 mod=CoStack [MEMORY] hold= 88768512 used= 88603392 count= 172 avg_used= 515136 mod=SUMMARY [2025-05-15 09:40:40.221463] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= KVSTORE_CACHE_ID hold= 140509184 used= 140509184 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 140509184 used= 0 count= 67 avg_used= 0 block_cnt= 0 chunk_cnt= 0 mod=KvstorCacheMb [MEMORY] hold= 140509184 used= 0 count= 67 avg_used= 0 mod=SUMMARY [2025-05-15 09:40:40.221480] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=11] [MEMORY] tenant_id= 1 ctx_id= META_OBJ_CTX_ID hold= 6291456 used= 5775360 limit= 1073741820 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 5373952 used= 5352960 count= 82 avg_used= 65280 block_cnt= 82 chunk_cnt= 3 mod=N_TabletPool [MEMORY] hold= 401408 used= 400064 count= 2 avg_used= 200032 block_cnt= 2 chunk_cnt= 1 mod=PoolFreeList [MEMORY] hold= 5775360 used= 5753024 count= 84 avg_used= 68488 mod=SUMMARY [2025-05-15 09:40:40.221505] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=19] [MEMORY] tenant_id= 1 ctx_id= TX_CALLBACK_CTX_ID hold= 4194304 used= 2869184 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 2869184 used= 2801408 count= 353 avg_used= 7936 block_cnt= 353 chunk_cnt= 2 mod=MemtableCallbac [MEMORY] hold= 2869184 used= 2801408 count= 353 avg_used= 7936 mod=SUMMARY [2025-05-15 09:40:40.221534] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= TX_DATA_TABLE hold= 37769216 used= 35215872 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 17862592 used= 17833792 count= 3 avg_used= 5944597 block_cnt= 3 chunk_cnt= 2 mod=MEMTABLE_ARENA [MEMORY] hold= 17345152 used= 16935424 count= 2134 avg_used= 7936 block_cnt= 2134 chunk_cnt= 10 mod=TX_DATA_SLICE [MEMORY] hold= 8128 used= 7936 count= 1 avg_used= 7936 block_cnt= 1 chunk_cnt= 1 mod=TX_DATA_ARENA [MEMORY] hold= 35215872 used= 34777152 count= 2138 avg_used= 16266 mod=SUMMARY [2025-05-15 09:40:40.221553] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=12] [MEMORY] tenant_id= 1 ctx_id= MDS_DATA_ID hold= 2097152 used= 186944 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 186944 used= 182528 count= 23 avg_used= 7936 block_cnt= 23 chunk_cnt= 1 mod=TX_OP [MEMORY] hold= 186944 used= 182528 count= 23 avg_used= 7936 mod=SUMMARY [2025-05-15 09:40:40.221576] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant_id= 1 ctx_id= MERGE_RESERVE_CTX_ID hold= 33554432 used= 0 limit= 9223372036854775807 [MEMORY] idle_size= 33554432 free_size= 33554432 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [2025-05-15 09:40:40.221598] INFO [LIB] operator() (ob_malloc_allocator.cpp:533) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=10] [MEMORY] tenant: 500, limit: 9223372036854775807 hold: 578543616 cache_hold: 0 cache_used: 0 cache_item_count: 0 [MEMORY] ctx_id= DEFAULT_CTX_ID hold_bytes= 186875904 limit= 9223372036854775807 [MEMORY] ctx_id= GLIBC hold_bytes= 18874368 limit= 9223372036854775807 [MEMORY] ctx_id= CO_STACK hold_bytes= 102760448 limit= 9223372036854775807 [MEMORY] ctx_id= LIBEASY hold_bytes= 4194304 limit= 9223372036854775807 [MEMORY] ctx_id= LOGGER_CTX_ID hold_bytes= 4194304 limit= 9223372036854775807 [MEMORY] ctx_id= PKT_NIO hold_bytes= 17969152 limit= 9223372036854775807 [MEMORY] ctx_id= SCHEMA_SERVICE hold_bytes= 11292672 limit= 9223372036854775807 [MEMORY] ctx_id= UNEXPECTED_IN_500 hold_bytes= 232382464 limit= 9223372036854775807 [2025-05-15 09:40:40.221901] INFO [LIB] print_usage (ob_tenant_ctx_allocator.cpp:173) [3980][MemoryDump][T0][Y0-0000000000000000-0-0] [lt=9] [MEMORY] tenant_id= 500 ctx_id= DEFAULT_CTX_ID hold= 186875904 used= 172036048 limit= 9223372036854775807 [MEMORY] idle_size= 0 free_size= 0 [MEMORY] wash_related_chunks= 0 washed_blocks= 0 washed_size= 0 [MEMORY] request_cached_chunk_cnt= 0 [MEMORY] hold= 31428608 used= 31368256 count= 7 avg_used= 4481179 block_cnt= 7 chunk_cnt= 6 mod=IoControl [MEMORY] hold= 12873728 used= 12856400 count= 1 avg_used= 12856400 block_cnt= 1 chunk_cnt= 1 mod=MemDumpContext [MEMORY] hold= 9457664 used= 9437784 count= 1 avg_used= 9437784 block_cnt= 1 chunk_cnt= 1 mod=HashBuckInteChe [MEMORY] hold= 8933376 used= 8912896 count= 1 avg_used= 8912896 block_cnt= 1 chunk_cnt= 1 mod=qdtable [MEMORY] hold= 7275200 used= 6334546 count= 1107 avg_used= 5722 block_cnt= 574 chunk_cnt= 6 mod=CharsetInit [MEMORY] hold= 5562368 used= 5551360 count= 40 avg_used= 138784 block_cnt= 37 chunk_cnt= 16 mod=TenantCtxAlloca [MEMORY] hold= 5533696 used= 5456720 count= 8 avg_used= 682090 block_cnt= 8 chunk_cnt= 4 mod=KvstCachWashStr [MEMORY] hold= 5283840 used= 5243040 count= 5 avg_used= 1048608 block_cnt= 5 chunk_cnt= 5 mod=ConcurHashMap [MEMORY] hold= 5242912 used= 5150064 count= 47 avg_used= 109575 block_cnt= 44 chunk_cnt= 12 mod=PartitTableTask [MEMORY] hold= 4747072 used= 4726520 count= 2 avg_used= 2363260 block_cnt= 2 chunk_cnt= 2 mod=HashBucTenComMo [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=PxP2PDhMgrKey [MEMORY] hold= 4739072 used= 4718712 count= 1 avg_used= 4718712 block_cnt= 1 chunk_cnt= 1 mod=HashPxBlooFilKe [MEMORY] hold= 4307456 used= 4278711 count= 3 avg_used= 1426237 block_cnt= 3 chunk_cnt= 3 mod=SqlDtlMgr [MEMORY] hold= 4232240 used= 4203008 count= 6 avg_used= 700501 block_cnt= 6 chunk_cnt= 5 mod=BaseLogWriter [MEMORY] hold= 4214896 used= 4194328 count= 2 avg_used= 2097164 block_cnt= 2 chunk_cnt= 2 mod=SerFuncRegHT [MEMORY] hold= 4194304 used= 4176267 count= 1 avg_used= 4176267 block_cnt= 1 chunk_cnt= 1 mod=SyslogCompress [MEMORY] hold= 3981312 used= 3903840 count= 12 avg_used= 325320 block_cnt= 12 chunk_cnt= 5 mod=DedupQueue [MEMORY] hold= 3829760 used= 3801088 count= 2 avg_used= 1900544 block_cnt= 2 chunk_cnt= 2 mod=TabletLSMap [MEMORY] hold= 2532032 used= 2513495 count= 3 avg_used= 837831 block_cnt= 3 chunk_cnt= 3 mod=FtParse [MEMORY] hold= 2383776 used= 2342304 count= 6 avg_used= 390384 block_cnt= 6 chunk_cnt= 5 mod=Omt [MEMORY] hold= 2379776 used= 2359608 count= 1 avg_used= 2359608 block_cnt= 1 chunk_cnt= 1 mod=HashBucIdUnitMa [MEMORY] hold= 2150400 used= 2113528 count= 3 avg_used= 704509 block_cnt= 3 chunk_cnt= 2 mod=FixedQueue [MEMORY] hold= 1835520 used= 1674496 count= 844 avg_used= 1984 block_cnt= 285 chunk_cnt= 22 mod=CreateContext [MEMORY] hold= 1581056 used= 1572904 count= 1 avg_used= 1572904 block_cnt= 1 chunk_cnt= 1 mod=IdConnMap [MEMORY] hold= 1522112 used= 1498368 count= 45 avg_used= 33297 block_cnt= 45 chunk_cnt= 16 mod=CommonArray [MEMORY] hold= 1286144 used= 1270296 count= 2 avg_used= 635148 block_cnt= 2 chunk_cnt= 2 mod=PxResMgr [MEMORY] hold= 1277952 used= 1260096 count= 3 avg_used= 420032 block_cnt= 3 chunk_cnt= 3 mod=CompEventMgr [MEMORY] hold= 1179648 used= 1048576 count= 16 avg_used= 65536 block_cnt= 16 chunk_cnt= 8 mod=[T]char [MEMORY] hold= 1073984 used= 1039504 count= 134 avg_used= 7757 block_cnt= 134 chunk_cnt= 3 mod=MallocInfoMap [MEMORY] hold= 1056768 used= 1048608 count= 1 avg_used= 1048608 block_cnt= 1 chunk_cnt= 1 mod=TsSourceInfoMap [MEMORY] hold= 1056768 used= 1011712 count= 6 avg_used= 168618 block_cnt= 6 chunk_cnt= 5 mod=LightyQueue [MEMORY] hold= 1056768 used= 1048608 count= 1 avg_used= 1048608 block_cnt= 1 chunk_cnt= 1 mod=ResRuleInfoMap [MEMORY] hold= 941872 used= 925368 count= 3 avg_used= 308456 block_cnt= 3 chunk_cnt= 3 mod=CACHE_INST_MAP [MEMORY] hold= 938576 used= 880440 count= 13 avg_used= 67726 block_cnt= 13 chunk_cnt= 8 mod=HashBucket [MEMORY] hold= 829632 used= 821248 count= 2 avg_used= 410624 block_cnt= 2 chunk_cnt= 2 mod=LSLocationMap [MEMORY] hold= 663552 used= 643296 count= 3 avg_used= 214432 block_cnt= 3 chunk_cnt= 3 mod=CompSuggestMgr [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=GEleTimer [MEMORY] hold= 647168 used= 640192 count= 1 avg_used= 640192 block_cnt= 1 chunk_cnt= 1 mod=EventTimer [MEMORY] hold= 633856 used= 619008 count= 78 avg_used= 7936 block_cnt= 78 chunk_cnt= 5 mod=ModulePageAlloc [MEMORY] hold= 622592 used= 619008 count= 1 avg_used= 619008 block_cnt= 1 chunk_cnt= 1 mod=SysTaskStatus [MEMORY] hold= 592320 used= 579328 count= 73 avg_used= 7936 block_cnt= 73 chunk_cnt= 14 mod=LinkHashNodeget_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-15 09:40:40.256344] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=45] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:40.258066] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.268176] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.271666] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=28] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:40.271695] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:40.271710] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:40.271725] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:40.271740] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=11][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:40.271751] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:40.271767] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:40.271778] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:40.271789] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:40.271800] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:40.271811] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:40.271824] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:40.271833] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:40.271849] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=16][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:40.271871] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:40.271883] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.271898] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.271910] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:40.271922] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:40.271936] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:40.271948] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:40.271971] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=18][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:40.271992] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:40.272003] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:40.272013] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:40.272055] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:40.272070] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4420][T1001_Occam][T1][YB427F000001-000635291DAC6241-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.272082] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4420][T1001_Occam][T1001][YB427F000001-000635291DAC6241-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:40.272096] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:40.272108] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:40.272120] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:40.272132] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302040271410, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:40.272145] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:40.272158] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1001, tenant_id=1, columns=0x7f1aa10563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:40.272177] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1001, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:40.272265] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1aa10563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:40.272283] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:40.272296] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:40.272308] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:40.272327] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:40.278651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.284757] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:338, tid:4304}]) [2025-05-15 09:40:40.289300] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.293736] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:40.293761] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.293773] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3977963, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.293790] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3977963, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.293827] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=36][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.293837] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3977963, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.293851] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.293867] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.293881] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.293896] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.293907] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.293922] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.293938] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040293936, replica_locations:[]}) [2025-05-15 09:40:40.293955] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.293975] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:304, local_retry_times:304, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:40.293996] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.294007] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:40.294023] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.294032] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.294040] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:40.294069] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:40.294083] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.294133] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11804184609, cache_obj->added_lc()=false, cache_obj->get_object_id()=15827, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.295361] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.295391] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.295427] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.295441] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.295456] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.295469] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.295478] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.295502] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.295517] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.295531] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.295543] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.295554] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.295567] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.295580] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040295579, replica_locations:[]}) [2025-05-15 09:40:40.295595] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.295608] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.295619] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.295634] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:40.295644] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:40.295654] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:40.295672] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:40.295685] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.295695] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.295705] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:40.295715] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:40.295724] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:40.295737] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:40.295748] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:40.295757] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:40.295766] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:40.295773] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:40.295783] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:40.295793] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:40.295813] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:40.295826] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.295841] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:40.295855] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:40.295872] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:40.295891] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=305, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:40.295929] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25] will sleep(sleep_us=100000, remain_us=3975753, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=305, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.299412] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.303439] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=23][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.303466] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.303496] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=29][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.303513] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=16][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.303524] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.303615] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=90][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.303631] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.303650] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.303667] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.303681] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.303698] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.303729] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.303747] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=14] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:40.305275] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:40.305299] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:40.305314] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:40.305329] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:40.305344] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:40.305352] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=8][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:40.305365] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:40.305376] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:40.305387] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:40.305399] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:40.305409] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:40.305419] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:40.305429] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:40.305439] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:40.305458] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:40.305471] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.305483] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.305503] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=18][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:40.305511] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=7][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:40.305524] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:40.305535] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:40.305551] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:40.305570] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=15][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:40.305579] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:40.305590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:40.305607] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:40.305621] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.305632] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:40.305644] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.305656] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:40.305669] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:40.305681] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302040305094, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.305693] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:40.305704] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.305768] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=12][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:40.305782] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=13][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:40.305795] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=13][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:40.305803] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=7][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:40.305815] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=10][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:40.305834] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=18][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:40.305845] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C2-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:40.306161] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.307107] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.307147] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040307100) [2025-05-15 09:40:40.307163] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302040107040, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.307197] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12342554992, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.307215] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.307239] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.307255] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.307263] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.307280] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12342556051, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.307293] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.307308] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.307321] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.307328] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.307347] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.307359] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.307371] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040307176) [2025-05-15 09:40:40.308270] WDIAG load_file_to_string (utility.h:673) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:40.308287] WDIAG get_ethernet_speed (utility.cpp:575) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:40.308300] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:40.308312] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3955][TimerWK2_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:40.309522] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.319008] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.319028] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.319043] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.319058] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.319070] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.319083] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.319095] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.319108] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.319119] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302040318992, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.319131] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.319148] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.319196] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.319208] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.319217] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.319230] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.319634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.324174] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.324197] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.324211] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.325378] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:40.325405] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.325421] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.325438] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:40.325448] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:40.325459] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:40.328768] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.328794] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040328762) [2025-05-15 09:40:40.328810] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302040128711, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.328838] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12342600567, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.328850] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12342428074, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.328860] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.328866] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.328880] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:40.328882] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:40.328897] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.328897] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.328905] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.328906] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.328919] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.328919] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.328930] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.328929] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.328936] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040328837) [2025-05-15 09:40:40.328937] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040328825) [2025-05-15 09:40:40.329741] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.336059] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:40.336079] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.336089] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.339852] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.344429] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:40.346342] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:164) [4291][T1_TimerWK2_MediumLoop][T1][YB427F000001-000635291D0C63C0-0-0] [lt=19] MediumCheckTask(cost_ts=9) [2025-05-15 09:40:40.349726] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5268, clean_start_pos=400000, clean_num=200000) [2025-05-15 09:40:40.349959] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.355449] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:40.355478] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.355503] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:40.355518] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:40.355525] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:40.355607] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=83][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:40.355622] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.355632] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:40.355642] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:40.355648] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:40.355702] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:40.360070] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.367095] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:40.367163] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:40.367231] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=21] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:40.367294] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:40.367323] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:11555, total:11565) [2025-05-15 09:40:40.367361] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:40.367410] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=44] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:11555, total:11565) [2025-05-15 09:40:40.367436] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:40.367456] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:40.367470] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:40.367501] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] successfully alloc block(macro_id={[ver=1,mode=0,seq=628775][2nd=1902]}) [2025-05-15 09:40:40.370184] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.380378] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=75][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.390502] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.395880] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=60] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.396017] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.396038] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=20] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.396084] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.396105] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3875620, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.396120] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3875620, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.396135] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.396144] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3875620, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.396159] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.396134] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=40] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.396171] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.396186] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.396199] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.396213] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.396224] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.396239] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.396258] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=18] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.396260] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:40.396271] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040396269, replica_locations:[]}) [2025-05-15 09:40:40.396288] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.396307] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:305, local_retry_times:305, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:40.396328] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.396339] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:40.396354] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.396364] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.396371] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:40.396389] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:40.396403] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.396459] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11804286934, cache_obj->added_lc()=false, cache_obj->get_object_id()=15828, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.397691] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.397720] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.397759] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.397774] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.397788] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.397802] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.397811] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.397823] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.397837] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.397850] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.397862] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.397872] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.397885] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.397898] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040397897, replica_locations:[]}) [2025-05-15 09:40:40.397913] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.397926] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.397937] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.397952] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:40.397962] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:40.397972] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:40.397989] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:40.398002] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.398011] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.398021] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:40.398030] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:40.398040] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:40.398051] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:40.398062] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:40.398071] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:40.398080] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:40.398089] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:40.398099] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:40.398107] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:40.398122] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:40.398132] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.398142] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:40.398152] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:40.398164] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:40.398172] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=306, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:40.398195] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] will sleep(sleep_us=100000, remain_us=3873486, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=306, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.400619] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.406221] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.406925] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4291][T1_TimerWK2_PalfGC][T1][Y0-0000000000000000-0-0] [lt=22] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}) [2025-05-15 09:40:40.407190] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12342655555, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.407215] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.407240] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.407257] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.407266] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.407282] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12342655555, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.407295] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.407309] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.407321] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.407331] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.407345] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.407353] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.407362] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040407164) [2025-05-15 09:40:40.410730] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.414622] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:40.414642] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.414654] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:40.414678] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4463][T1001_TimerWK3_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:40.419176] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.419197] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.419220] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.419237] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.419250] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.419263] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.419277] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.419289] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.419304] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302040419143, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.419318] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.419328] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.419412] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.419424] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.419435] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.419446] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.420839] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.424231] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.424250] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.424265] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.425433] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:40.425460] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.425477] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.425503] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:40.425515] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:40.425527] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:40.428835] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=12342700017, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.428856] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.428879] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:40.428893] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.428893] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:40.428907] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.428910] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.428923] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.428931] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.428930] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040428888) [2025-05-15 09:40:40.428940] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040428819) [2025-05-15 09:40:40.428944] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302040228837, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.428965] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12342528580, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.428978] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.428993] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:40.429005] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.429014] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.429026] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.429034] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.429042] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040428955) [2025-05-15 09:40:40.430950] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.436120] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=95][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:40.436138] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.436151] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.441019] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.446934] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:40.451129] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.462260] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.467426] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=32][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.467444] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.467458] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:40.467479] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:40.472371] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.482479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.492598] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.498329] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.498352] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3773373, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.498370] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3773373, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.498381] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.498393] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3773373, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.498408] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.498425] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.498439] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.498451] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.498460] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.498475] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.498501] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040498499, replica_locations:[]}) [2025-05-15 09:40:40.498519] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.498538] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:306, local_retry_times:306, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:40.498560] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.498572] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:40.498588] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.498600] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.498607] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:40.498637] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:40.498651] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.498700] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11804389177, cache_obj->added_lc()=false, cache_obj->get_object_id()=15829, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.499847] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.499875] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.499915] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.499930] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.499945] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.499958] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.499965] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=5][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.499978] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.499993] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.500008] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.500021] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.500033] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.500047] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.500058] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040500057, replica_locations:[]}) [2025-05-15 09:40:40.500081] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.500104] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.500133] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.500150] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:40.500161] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:40.500173] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:40.500191] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:40.500205] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.500212] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.500225] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:40.500235] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:40.500242] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:40.500255] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:40.500268] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:40.500279] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:40.500289] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:40.500300] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:40.500310] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:40.500322] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:40.500338] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:40.500350] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.500361] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:40.500374] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:40.500388] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:40.500398] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=307, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:40.500423] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] will sleep(sleep_us=100000, remain_us=3771259, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=307, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.502714] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.506281] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.506882] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:40.506910] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.506928] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:40.506952] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:40.506964] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:40.506974] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:40.506986] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:40.507001] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:40.507109] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.507130] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.507145] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:40.507159] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:40.507175] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:40.507190] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:40.507212] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:40.507229] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:40.507226] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.507252] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040507218) [2025-05-15 09:40:40.507267] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:40.507275] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302040307176, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.507316] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12342755909, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.507337] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.507362] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.507378] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.507390] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.507406] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12342755909, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.507418] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.507431] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.507442] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.507452] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.507464] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.507473] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.507483] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.512822] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.519205] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.519225] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.519246] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.519263] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.519276] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.519288] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.519298] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.519309] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.519320] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302040519172, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.519333] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.519343] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.519425] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.519439] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.519449] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.519460] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.522939] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.524290] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.524308] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.524323] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.525505] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:40.525527] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.525539] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.525555] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:40.525565] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:40.525577] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:40.528881] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.528904] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040528877) [2025-05-15 09:40:40.528919] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302040328825, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.528949] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12342800409, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.528958] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12342627916, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.528965] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.528976] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.528984] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.528993] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.533051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.536178] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:40.536199] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.536213] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:40.536227] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302040535655, register_time_us=-1, curr_time_us=1747302040536225) [2025-05-15 09:40:40.536245] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:40.536255] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.542418] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=21] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:40.543165] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.544333] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=15] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:40.549981] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:40.551315] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4634][T1002_ArcSender][T1002][YB427F000001-000635291E2C6112-0-0] [lt=22][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:40.553275] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.563387] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.570059] INFO pn_ratelimit (group.c:653) [3955][TimerWK2_IngressService][T0][Y0-0000000000000000-0-0] [lt=10] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-15 09:40:40.571075] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.571097] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.571124] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:40.571137] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:40.571152] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:40.571169] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] refresh gts failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:40.571180] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:40.571188] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:40.571196] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.571219] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:40.571230] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:40.571242] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:40.571252] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:40.571259] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:40.571266] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:40.571275] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:40.571630] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:40.571644] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:40.571660] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.571669] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:40.571678] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:40.571686] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:40.571695] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:40.571704] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.571718] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:40.573500] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.579230] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6695-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.579256] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6695-0-0] [lt=25][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.579281] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6695-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.579296] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4577][T1002_TimerWK0_LSMetaCh][T1001][YB427F000001-000635291FCC6695-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.579306] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=8][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.579319] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.579331] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.579341] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.579351] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302040579203, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.579364] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.579375] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:40.579441] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=10][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:40.579453] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=11][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:40.579464] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=10][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:40.579474] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=9][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:40.579486] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=9][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:40.579506] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=17][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:40.579515] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4577][T1002_TimerWK0_LSMetaCh][T1002][YB427F000001-000635291FCC6695-0-0] [lt=9][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:40.582321] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4580][T1002_TimerWK3_TenantMetaMemMg][T1002][Y0-0000000000000000-0-0] [lt=20] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:728, free_obj_hold_cnt:0, allocator used:2795520, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=3002880, tablets_mem_limit=7774753240, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:9, free_obj_hold_cnt:0, allocator used:18432, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=728) [2025-05-15 09:40:40.583615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.586523] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553198, support_ssl=false) [2025-05-15 09:40:40.586598] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=39] sm conn init succ(conn.sessid_=3221553198, sess.client_addr_="172.22.0.1:39982") [2025-05-15 09:40:40.586633] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553198, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.586831] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.586871] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.586898] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.586927] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.586943] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.586956] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.586979] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.586995] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553198, proxy_sessid=0) [2025-05-15 09:40:40.587024] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=9][errcode=0] server close connection(sessid=3221553198, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.587037] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553198, proxy_sessid=0) [2025-05-15 09:40:40.587045] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.587058] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D1-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553198, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.587114] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553198, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040586312, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.587141] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553198, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.587155] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553198, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040586312, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.587175] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553198) [2025-05-15 09:40:40.587191] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553198, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.593726] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.600547] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.600572] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3671149, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.600588] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3671149, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.600602] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.600611] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3671149, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.600626] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.600643] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.600657] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.600672] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.600684] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.600700] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.600719] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040600716, replica_locations:[]}) [2025-05-15 09:40:40.600742] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.600761] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:307, local_retry_times:307, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:40.600782] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.600800] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:40.600812] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.600823] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:40.600830] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:40.600848] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:40.600862] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.600903] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11804491383, cache_obj->added_lc()=false, cache_obj->get_object_id()=15830, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.601752] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [3953][TimerWK0_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.601773] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3953][TimerWK0_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.601786] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3953][TimerWK0_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to get percent [2025-05-15 09:40:40.601796] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3953][TimerWK0_SqlMemTimer][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to calculate global bound size(ret=-5627, ids[i]=1001) [2025-05-15 09:40:40.601817] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [3953][TimerWK0_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.601829] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [3953][TimerWK0_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:40.601838] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [3953][TimerWK0_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:40.601845] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [3953][TimerWK0_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:40.601853] WDIAG [SQL.ENG] get_max_work_area_size (ob_tenant_sql_memory_manager.cpp:838) [3953][TimerWK0_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:40.601862] WDIAG [SQL.ENG] calculate_global_bound_size (ob_tenant_sql_memory_manager.cpp:1230) [3953][TimerWK0_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] failed to get percent [2025-05-15 09:40:40.601869] WDIAG [SERVER] runTimerTask (ob_tenant_duty_task.cpp:205) [3953][TimerWK0_SqlMemTimer][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] failed to calculate global bound size(ret=-5157, ids[i]=1002) [2025-05-15 09:40:40.601943] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.601969] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.602007] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.602022] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.602036] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.602050] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.602058] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.602071] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.602087] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:40.602100] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.602115] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.602123] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.602138] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.602153] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040602152, replica_locations:[]}) [2025-05-15 09:40:40.602170] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.602183] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.602195] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:40.602211] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:40.602222] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:40.602234] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:40.602251] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:40.602265] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.602272] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:40.602280] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:40.602291] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:40.602303] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:40.602314] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:40.602326] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:40.602333] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:40.602344] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:40.602354] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:40.602365] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:40.602376] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:40.602391] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:40.602402] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:40.602415] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:40.602426] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:40.602441] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:40.602448] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=308, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:40.602473] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=3669208, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=308, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.603846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.606340] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.607305] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19] get wrs ts(ls_id={id:1}, delta=12342855187, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.607333] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.607366] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12342855187, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.607379] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.608800] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=93] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:40.608832] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=1] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:40.613964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.619180] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.619201] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.619220] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.619236] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.619248] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.619262] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.619274] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.619285] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.619295] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302040619159, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.619308] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.619316] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.619375] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.619386] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.619397] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.619407] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.624349] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:40.624370] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.625573] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.626277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.628948] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12342900782, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.628969] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.629010] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.629050] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040629003) [2025-05-15 09:40:40.629064] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302040428954, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.629087] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12342728289, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.629100] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.630132] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553200, support_ssl=false) [2025-05-15 09:40:40.630181] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221553200, sess.client_addr_="172.22.0.1:39984") [2025-05-15 09:40:40.630198] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553200, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.630200] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553201, support_ssl=false) [2025-05-15 09:40:40.630327] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553201, sess.client_addr_="172.22.0.1:39994") [2025-05-15 09:40:40.630344] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553201, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.630508] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.630539] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.630561] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=36][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.630586] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.630595] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.630607] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.630625] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.630632] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553202, support_ssl=false) [2025-05-15 09:40:40.630639] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553200, proxy_sessid=0) [2025-05-15 09:40:40.630662] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=9][errcode=0] server close connection(sessid=3221553200, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.630667] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553202, sess.client_addr_="172.22.0.1:40006") [2025-05-15 09:40:40.630674] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553200, proxy_sessid=0) [2025-05-15 09:40:40.630682] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:40.630679] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553202, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.630692] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A2-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553200, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.630718] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.630743] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553200, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302040629674, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.630758] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.630766] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553200, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.630779] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553200, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302040629674, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.630797] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553200) [2025-05-15 09:40:40.630801] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.630843] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=39][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.630865] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=21][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.630876] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.630911] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=21] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.630900] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553200, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.630937] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=22][errcode=-4018] get session fail(ret=-4018, sessid=3221553201, proxy_sessid=0) [2025-05-15 09:40:40.630952] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.630973] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.630986] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=22][errcode=0] server close connection(sessid=3221553201, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.630990] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.631005] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.631013] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.631007] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=21][errcode=-4018] get session fail(ret=-4018, sessid=3221553201, proxy_sessid=0) [2025-05-15 09:40:40.631019] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.631038] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.631036] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=28][errcode=-4016] session is null [2025-05-15 09:40:40.631048] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553202, proxy_sessid=0) [2025-05-15 09:40:40.631070] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=11][errcode=0] server close connection(sessid=3221553202, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.631082] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553202, proxy_sessid=0) [2025-05-15 09:40:40.631061] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9091-0-0] [lt=22] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553201, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.631089] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:40.631096] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D2-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553202, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.631136] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553201, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302040629674, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.631169] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553202, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040630733, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.631189] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553201, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.631202] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221553202, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.631211] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553201, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302040629674, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.631227] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553201) [2025-05-15 09:40:40.631244] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553201, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.631272] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553202, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040630733, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.631286] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553202) [2025-05-15 09:40:40.631295] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553202, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.631634] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553203, support_ssl=false) [2025-05-15 09:40:40.631738] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=38] sm conn init succ(conn.sessid_=3221553203, sess.client_addr_="172.22.0.1:40012") [2025-05-15 09:40:40.631758] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553203, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.631956] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.631992] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.632004] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.632019] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.632026] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.632038] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.632050] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.632063] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553203, proxy_sessid=0) [2025-05-15 09:40:40.632080] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=7][errcode=0] server close connection(sessid=3221553203, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.632091] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553203, proxy_sessid=0) [2025-05-15 09:40:40.632098] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.632104] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A3-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553203, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.632152] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553203, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040631792, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.632174] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553203, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.632183] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553203, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040631792, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.632200] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553203) [2025-05-15 09:40:40.632223] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221553203, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.632252] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553204, support_ssl=false) [2025-05-15 09:40:40.632353] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=38] sm conn init succ(conn.sessid_=3221553204, sess.client_addr_="172.22.0.1:40018") [2025-05-15 09:40:40.632373] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553204, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.632569] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.632610] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=23] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.632632] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=34][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.632644] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221553205, support_ssl=false) [2025-05-15 09:40:40.632654] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.632664] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.632677] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.632682] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221553205, sess.client_addr_="172.22.0.1:40024") [2025-05-15 09:40:40.632692] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.632694] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553205, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.632708] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553204, proxy_sessid=0) [2025-05-15 09:40:40.632728] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=8][errcode=0] server close connection(sessid=3221553204, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.632742] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553204, proxy_sessid=0) [2025-05-15 09:40:40.632749] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.632765] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9092-0-0] [lt=15] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553204, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.632838] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553204, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302040631792, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.632864] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553204, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.632870] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.632876] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553204, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302040631792, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.632891] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553204) [2025-05-15 09:40:40.632895] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=9] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.632906] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553204, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.632916] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.632941] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.632957] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.632965] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.632987] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.633003] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553205, proxy_sessid=0) [2025-05-15 09:40:40.633024] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=8][errcode=0] server close connection(sessid=3221553205, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.633037] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553205, proxy_sessid=0) [2025-05-15 09:40:40.633044] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.633056] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D3-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553205, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.633079] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221553206, support_ssl=false) [2025-05-15 09:40:40.633169] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553206, sess.client_addr_="172.22.0.1:40028") [2025-05-15 09:40:40.633188] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553206, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633220] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553205, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302040632852, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633239] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553205, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.633246] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553205, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302040632852, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.633268] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553205) [2025-05-15 09:40:40.633285] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553205, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.633357] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553207, support_ssl=false) [2025-05-15 09:40:40.633367] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.633392] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.633402] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221553207, sess.client_addr_="172.22.0.1:40044") [2025-05-15 09:40:40.633414] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.633418] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553207, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633434] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.633442] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.633453] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.633467] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.633482] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553206, proxy_sessid=0) [2025-05-15 09:40:40.633523] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=29][errcode=0] server close connection(sessid=3221553206, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.633535] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553206, proxy_sessid=0) [2025-05-15 09:40:40.633542] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:40.633549] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A4-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553206, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.633587] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.633602] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553206, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302040632852, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633618] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.633622] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553206, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.633634] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553206, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302040632852, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.633649] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553206) [2025-05-15 09:40:40.633641] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=34][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.633662] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.633662] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553206, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.633679] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.633687] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.633701] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.633718] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553207, proxy_sessid=0) [2025-05-15 09:40:40.633710] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=41] new mysql sessid created(conn.sessid_=3221553208, support_ssl=false) [2025-05-15 09:40:40.633738] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=8][errcode=0] server close connection(sessid=3221553207, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.633752] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553207, proxy_sessid=0) [2025-05-15 09:40:40.633760] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:40.633760] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221553208, sess.client_addr_="172.22.0.1:40050") [2025-05-15 09:40:40.633769] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9093-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553207, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.633777] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553208, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633817] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553209, support_ssl=false) [2025-05-15 09:40:40.633820] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553207, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302040632852, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633841] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553207, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.633852] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553207, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302040632852, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.633865] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221553209, sess.client_addr_="172.22.0.1:40064") [2025-05-15 09:40:40.633870] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553207) [2025-05-15 09:40:40.633878] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553209, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633886] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553207, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.633929] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553210, support_ssl=false) [2025-05-15 09:40:40.633964] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553210, sess.client_addr_="172.22.0.1:40066") [2025-05-15 09:40:40.633978] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553210, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.633982] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.634023] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.634040] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.634058] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.634074] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.634082] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.634100] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.634115] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553208, proxy_sessid=0) [2025-05-15 09:40:40.634085] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.634132] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=7][errcode=0] server close connection(sessid=3221553208, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.634143] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553208, proxy_sessid=0) [2025-05-15 09:40:40.634143] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=45] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.634142] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:40.634150] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.634157] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D4-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553208, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.634163] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:40.634161] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.634178] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:40.634185] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.634192] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=14][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:40.634195] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.634206] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.634204] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:40.634215] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:40.634204] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553208, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040633922, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.634227] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.634224] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.634230] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553208, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.634238] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553210, proxy_sessid=0) [2025-05-15 09:40:40.634239] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553209, proxy_sessid=0) [2025-05-15 09:40:40.634240] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553208, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302040633922, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.634258] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553208) [2025-05-15 09:40:40.634259] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=11][errcode=0] server close connection(sessid=3221553210, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.634270] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553210, proxy_sessid=0) [2025-05-15 09:40:40.634278] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:40.634277] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553208, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.634273] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=15][errcode=0] server close connection(sessid=3221553209, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:40.634291] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9094-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553210, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.634298] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=25][errcode=-4018] get session fail(ret=-4018, sessid=3221553209, proxy_sessid=0) [2025-05-15 09:40:40.634310] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:40.634320] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A5-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553209, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:40.634336] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553210, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302040633922, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.634354] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553210, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.634361] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553210, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302040633922, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.634377] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553210) [2025-05-15 09:40:40.634376] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553209, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302040633922, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:40.634390] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553210, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.634397] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553209, proxy_sessid=0, ret=0) [2025-05-15 09:40:40.634410] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553209, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302040633922, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:40.634432] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221553209) [2025-05-15 09:40:40.634442] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553209, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:40.636238] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.636390] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.636641] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10] run rewrite rule refresh task(rule_mgr_->tenant_id_=1001) [2025-05-15 09:40:40.637996] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][YB427F000001-000635291D5C6446-0-0] [lt=13][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.638026] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4461][T1001_TimerWK1_ReqMemEvict][T1001][YB427F000001-000635291D5C6446-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.646500] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.647028] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:40.656613] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.661121] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EB023-0-0] [lt=37][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1001, ls_id:{id:1}, add_timestamp:1747300526118711}) [2025-05-15 09:40:40.666725] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.676833] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.685587] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=23][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:35, tid:4630}]) [2025-05-15 09:40:40.686941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.697048] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.702631] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.702657] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3569073, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.702763] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=106][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3569073, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.702778] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:40.702789] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3569073, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:40.702806] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:40.702824] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.702841] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.702855] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.702869] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.702886] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.702902] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040702899, replica_locations:[]}) [2025-05-15 09:40:40.702923] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.702947] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.704406] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.704431] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.704454] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.704473] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.704497] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.704510] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.704525] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.704540] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040704539, replica_locations:[]}) [2025-05-15 09:40:40.704611] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=3567071, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=309, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.706402] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.707158] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.707343] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.707373] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040707336) [2025-05-15 09:40:40.707389] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302040507289, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.707424] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12342955777, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.707444] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.707467] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.707485] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.707508] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.707526] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12342955777, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.707544] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.707559] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.707570] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.707580] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.707593] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.707602] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.707611] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040707402) [2025-05-15 09:40:40.717267] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.724419] INFO [PALF] check_parent_health (log_config_mgr.cpp:2875) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=1] re_register_parent reason(palf_id=1, self="127.0.0.1:2882", is_registering_timeout=false, first_registration=true, parent_timeout=false, parent_keepalive_time_us=-1, last_submit_register_req_time_us=-1, last_first_register_time_us=1747302040723904, register_time_us=-1, curr_time_us=1747302040724417) [2025-05-15 09:40:40.724454] WDIAG [PALF] check_parent_health (log_config_mgr.cpp:2882) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4023] register request timeout, re_register_parent failed(ret=-4023, ret="OB_EAGAIN", palf_id=1, self="127.0.0.1:2882") [2025-05-15 09:40:40.725615] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0] LogLoopThread round_cost_time(us)(round_cost_time=76) [2025-05-15 09:40:40.727376] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.728996] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.729031] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040728990) [2025-05-15 09:40:40.729047] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302040528933, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.729075] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12343000304, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.729077] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=1] get wrs ts(ls_id={id:1}, delta=12342827811, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.729093] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.729103] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.729117] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:40.729121] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:40.729132] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.729137] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.729141] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.729145] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.729154] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.729156] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.729162] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.729164] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.729171] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040729062) [2025-05-15 09:40:40.729169] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040729058) [2025-05-15 09:40:40.738983] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.742067] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3955][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=19] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:40.749093] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.750262] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=21] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:40.756392] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.756414] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:40.756425] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:40.756447] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:40.756462] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:40.756499] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:40.759208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.769319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.779429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.785810] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:29, tid:4602}]) [2025-05-15 09:40:40.789544] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.799656] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.800788] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=22] ====== tenant freeze timer task ====== [2025-05-15 09:40:40.804764] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.804796] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3466941, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.804824] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3466941, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.804854] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.804871] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.804885] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.804902] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.804921] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040804919, replica_locations:[]}) [2025-05-15 09:40:40.804941] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.804967] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.806382] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.806404] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.806420] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.806442] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.806457] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.806462] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.806467] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.806481] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.806503] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040806502, replica_locations:[]}) [2025-05-15 09:40:40.806574] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=3465108, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=310, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.807415] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12343055200, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.807445] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.807472] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.807508] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=34][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.807523] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.807540] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12343056263, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.807553] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.807567] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.807578] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.807588] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.807601] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.807610] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.807619] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040807393) [2025-05-15 09:40:40.809766] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.819356] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.819380] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.819400] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.819421] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.819434] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.819448] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.819459] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.819469] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302040819327, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.819483] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.819505] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.819583] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.819597] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.819608] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.819617] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.819882] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.820247] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:40.827552] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4461][T1001_TimerWK1_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-05-15 09:40:40.827571] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4461][T1001_TimerWK1_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=16] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:40.829061] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12343100794, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.829080] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.829101] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:40.829115] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.829132] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.829128] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.829148] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.829156] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.829163] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040829049) [2025-05-15 09:40:40.829160] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040829119) [2025-05-15 09:40:40.829178] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302040629076, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.829209] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12342928301, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.829225] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.829245] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:40.829266] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.829281] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.829297] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.829309] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.829322] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040829197) [2025-05-15 09:40:40.829991] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.840097] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.847121] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:40.850208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.860319] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.860652] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=30] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:40.861186] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4243][T1_Occam][T1][YB427F000001-000635291B4C65D3-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.870426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.880537] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.890653] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.900765] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.906522] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:40.906701] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.906723] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3364997, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.906740] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3364997, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.906764] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.906778] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.906791] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.906806] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.906822] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040906820, replica_locations:[]}) [2025-05-15 09:40:40.906840] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:40.906863] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:40.907463] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.907521] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=44][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040907453) [2025-05-15 09:40:40.907539] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302040707401, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.907583] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12343155779, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.907650] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=63] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.907679] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.907701] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.907712] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.907728] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12343155779, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:40.907741] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.907755] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:40.907764] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:40.907772] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:40.907786] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.907798] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:40.907811] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040907557) [2025-05-15 09:40:40.908048] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.908069] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.908085] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.908108] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.908121] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.908129] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.908146] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.908162] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040908161, replica_locations:[]}) [2025-05-15 09:40:40.908226] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=3363456, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=311, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:40.910893] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=29][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.919458] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.919480] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.919524] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.919544] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.919558] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.919573] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.919584] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.919595] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302040919423, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.919611] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.919622] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:40.919707] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:40.919718] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:40.919728] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:40.919738] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:40.921011] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.922654] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.922667] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:40.922675] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=7] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1001) [2025-05-15 09:40:40.922687] WDIAG [SQL.PC] run_plan_cache_task (ob_plan_cache.cpp:2766) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] fail to update plan cache memory sys val(ret=-4029) [2025-05-15 09:40:40.922700] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=8] start lib cache evict(tenant_id=1001, mem_hold=2097152, mem_limit=215965365, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:40.922719] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=17] end lib cache evict(tenant_id=1001, cache_evict_num=0, mem_hold=2097152, mem_limit=215965365, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:40.922730] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=9] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:40.925344] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=9] Dumping All Cache Objs(alloc_obj_list.count()=0, alloc_obj_list=[]) [2025-05-15 09:40:40.925363] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4460][T1001_TimerWK0_PlanCacheEvict][T1001][Y0-0000000000000000-0-0] [lt=18] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:40.926658] INFO [STORAGE.TRANS] run1 (ob_xa_trans_heartbeat_worker.cpp:87) [4627][T1002_ObXAHbWor][T1002][Y0-0000000000000000-0-0] [lt=26] XA scheduler heartbeat task statistics(avg_time=1) [2025-05-15 09:40:40.928333] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=30] PNIO [ratelimit] time: 1747302040928331, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-05-15 09:40:40.928351] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=26][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.928368] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=16][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.928384] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.928400] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.928412] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.928426] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:40.928437] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.928446] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.928421] INFO [ARCHIVE] do_thread_task_ (ob_archive_fetcher.cpp:287) [4500][T1001_ArcFetche][T1001][YB427F000001-000635291CAC6112-0-0] [lt=29] ObArchiveFetcher is running(thread_index=0) [2025-05-15 09:40:40.928457] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302040928336, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:40.928468] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.928477] WDIAG [SHARE] get_valid_dest_pairs (ob_archive_persist_helper.cpp:427) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=8][errcode=-5627] failed to exec sql(ret=-5627, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:40.928541] WDIAG [ARCHIVE] load_archive_round_attr (ob_archive_persist_mgr.cpp:294) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=22][errcode=-5627] get valid dest pair failed(ret=-5627, tenant_id_=1002) [2025-05-15 09:40:40.928554] WDIAG [ARCHIVE] do_check_switch_archive_ (ob_archive_service.cpp:254) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=12][errcode=-5627] load archive round attr failed(ret=-5627, tenant_id=1002) [2025-05-15 09:40:40.929112] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:40.929130] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302040929111, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302040929111, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:40.929172] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=38][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:40.929190] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302040929167) [2025-05-15 09:40:40.929194] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12343027823, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:40.929204] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302040729058, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:40.929213] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.929228] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12343201375, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:40.929237] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:40.929243] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:40.929252] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.929260] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.929257] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:40.929268] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:40.929271] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.929275] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:40.929279] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:40.929285] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.929286] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040929175) [2025-05-15 09:40:40.929293] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:40.929299] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302040929216) [2025-05-15 09:40:40.929714] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:40.929725] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=31] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:40.929893] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.929910] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=16][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.929924] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4636][T1002_ArcSrv][T1001][YB427F000001-000635291E5C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.929942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=1][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.929957] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:40.929970] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.929983] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.929996] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302040929883, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:40.930010] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.930022] WDIAG [SHARE] get_valid_dest_pairs (ob_archive_persist_helper.cpp:427) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=12][errcode=-5627] failed to exec sql(ret=-5627, sql=select dest_no, value from __all_log_archive_dest_parameter where tenant_id=1002 and name='dest_id' order by dest_no asc) [2025-05-15 09:40:40.930067] WDIAG [ARCHIVE] load_archive_round_attr (ob_archive_persist_mgr.cpp:294) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=12][errcode=-5627] get valid dest pair failed(ret=-5627, tenant_id_=1002) [2025-05-15 09:40:40.930080] WDIAG [ARCHIVE] persist_archive_progress_ (ob_archive_persist_mgr.cpp:341) [4636][T1002_ArcSrv][T1002][YB427F000001-000635291E5C6112-0-0] [lt=11][errcode=-5627] load archive round attr failed(ret=-5627, attr={key:{tenant_id:0, dest_no:-1}, incarnation:1, dest_id:0, round_id:0, state:{status:"INVALID"}, start_scn:{val:0, v:0}, checkpoint_scn:{val:0, v:0}, max_scn:{val:0, v:0}, compatible:{version:1}, base_piece_id:0, used_piece_id:0, piece_switch_interval:0, frozen_input_bytes:0, frozen_output_bytes:0, active_input_bytes:0, active_output_bytes:0, deleted_input_bytes:0, deleted_output_bytes:0, path:"", comment:""}) [2025-05-15 09:40:40.931124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.934220] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=21] PNIO [ratelimit] time: 1747302040934216, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007619, add_bytes: 0 [2025-05-15 09:40:40.936788] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=12] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4844059054, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:40.936890] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=29] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:40.936906] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=16] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4844059054, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:40.937447] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.937472] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.937499] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=28][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.937514] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.937533] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.937548] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.937560] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.937576] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.937588] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.937604] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:40.937616] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:40.937623] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=6][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:40.937644] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:40.937656] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.937669] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.937687] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.937699] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:40.937710] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:40.937723] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:40.937742] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=14] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302040937740, replica_locations:[]}) [2025-05-15 09:40:40.937765] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:40.937818] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:40.937834] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:40.937870] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=35] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:40.937892] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:40.937914] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=22] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=119, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:40.937937] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=23][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:40.937961] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=22][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:40.937971] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:40.941235] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.950548] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=7] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:40.951348] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.961457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.962000] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=27] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:40.964606] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4531][T1002_Occam][T1002][YB427F000001-000635291EDC65CF-0-0] [lt=11] for each ls(succ_num=2, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.965578] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=101] tx gc loop thread is running(MTL_ID()=1) [2025-05-15 09:40:40.965595] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=16] try gc retain ctx [2025-05-15 09:40:40.965601] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=6] try to retry start_working [2025-05-15 09:40:40.965632] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=9] finish update min start scn(max_decided_scn={val:1747289697727835001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289694525675001, v:0}, keep_alive_scn_:{val:1747289694525675001, v:0}, update_ts_:1747302040965197}) [2025-05-15 09:40:40.966225] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4340][T1_TxLoopWorker][T1][Y0-0000000000000000-0-0] [lt=21] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:40.970500] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=13] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:40.970534] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=23] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:40.970552] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=16] [tabletchange] task check ls(ls->get_ls_id()={id:1001}, tablet_persist_trigger=0) [2025-05-15 09:40:40.970565] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4579][T1002_TimerWK2_][T1002][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=77) [2025-05-15 09:40:40.971577] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.972221] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:40.972258] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5323) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=0] is_in_sync(ret=-4721, palf_id=1, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1747289697927760000, v:0}) [2025-05-15 09:40:40.974852] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [4289][T1_TimerWK0_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=7] clear dtl interm result cost(us)(clear_cost=2629, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-15 09:40:40.981688] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.986221] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:43, tid:3961}]) [2025-05-15 09:40:40.991795] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:40.995944] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE8-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.995962] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE8-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.995980] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE8-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.995997] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE8-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.996011] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE8-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.996024] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:40.996037] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.996049] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.996060] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302040995914, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:40.996071] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.996080] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:40.996159] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:40.997404] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE9-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.997421] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE9-0-0] [lt=16][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:40.997431] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE9-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:40.997446] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CE9-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:40.997453] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CE9-0-0] [lt=7][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:40.997465] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:40.997475] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:40.997485] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:40.997506] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302040997396, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:40.997518] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:40.997525] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:40.997565] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:40.998922] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:41.001901] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.004989] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553219, support_ssl=false) [2025-05-15 09:40:41.005039] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] sm conn init succ(conn.sessid_=3221553219, sess.client_addr_="127.0.0.1:51838") [2025-05-15 09:40:41.005058] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553219, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.005148] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.005187] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.005211] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.005235] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.005250] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.005264] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.005286] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.005304] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553219, proxy_sessid=0) [2025-05-15 09:40:41.005330] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=9][errcode=0] server close connection(sessid=3221553219, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.005343] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553219, proxy_sessid=0) [2025-05-15 09:40:41.005352] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:41.005365] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D5-0-0] [lt=11] MySQL LOGIN(direct_client_ip="127.0.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553219, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=3908101, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.005412] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553219, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041004323, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:118, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.005435] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553219, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.005451] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553219, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041004323, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:118, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.005479] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=26] mark session id unused(conn.sessid_=3221553219) [2025-05-15 09:40:41.005613] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=35] connection close(sessid=3221553219, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.006586] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.007082] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.007324] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=46][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:41.007344] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:41.007362] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:41.007375] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:41.007385] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:41.007398] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:41.007416] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:41.007430] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:41.007469] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:41.007531] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12343256300, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.007556] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.007580] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.007600] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.007612] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.007633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12343256300, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.007646] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.007661] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.007673] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.007681] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.007694] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.007706] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.007716] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041007504) [2025-05-15 09:40:41.008362] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.008385] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3263336, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.008405] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3263336, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.008422] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.008438] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3263336, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.008457] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.008479] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.008522] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=36][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.008542] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.008557] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.008586] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=25] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.008606] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041008603, replica_locations:[]}) [2025-05-15 09:40:41.008624] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.008655] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.008669] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:41.008699] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.008752] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11804899227, cache_obj->added_lc()=false, cache_obj->get_object_id()=15834, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.010049] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.010070] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.010090] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.010106] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.010121] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.010137] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.010157] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.010174] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.010190] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.010204] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.010222] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.010239] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041010238, replica_locations:[]}) [2025-05-15 09:40:41.010313] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=3261369, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=312, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.012010] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.014797] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4326][T1_LCLSender][T1][Y0-0000000000000000-0-0] [lt=54] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1ac33de2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:41.019569] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.019590] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.019612] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.019630] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.019643] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.019656] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.019669] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.019684] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.019695] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302041019537, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.019709] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.019721] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.019796] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:41.019809] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:41.019818] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:41.019828] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:41.022117] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.023346] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4577][T1002_TimerWK0_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=0] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:41.023372] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4577][T1002_TimerWK0_TabletShell][T1002][Y0-0000000000000000-0-0] [lt=19] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=2, times=83) [2025-05-15 09:40:41.029173] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:41.029209] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=12343300779, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.029229] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.029239] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.029249] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.029263] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.029265] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041029232) [2025-05-15 09:40:41.029271] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.029280] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302040829192, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.029286] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.029298] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.029301] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12343128286, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.029305] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041029197) [2025-05-15 09:40:41.029315] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.029341] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.029354] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.029365] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.029378] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.029388] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.029394] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041029291) [2025-05-15 09:40:41.032227] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.042345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.047214] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:41.052457] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.056852] INFO [CLOG] operator() (ob_log_replay_service.cpp:1671) [4463][T1001_TimerWK3_ReplayProcessSt][T1001][Y0-0000000000000000-0-0] [lt=12] get_replay_process success(id={id:1}, submitted_log_size=144320102, unsubmitted_log_size=144320102, replayed_log_size=144320102, unreplayed_log_size=0) [2025-05-15 09:40:41.056892] INFO [CLOG] runTimerTask (ob_log_replay_service.cpp:174) [4463][T1001_TimerWK3_ReplayProcessSt][T1001][Y0-0000000000000000-0-0] [lt=22] dump tenant replay process(tenant_id=1001, unsubmitted_log_size(MB)=137, unreplayed_log_size(MB)=0, estimate_time(second)=0, submitted_log_size(MB)=137, replayed_log_size(MB)=137, last_submitted_log_size(MB)=137, last_replayed_log_size(MB)=137, round_cost_time(second)=10, pending_replay_log_size(MB)=0) [2025-05-15 09:40:41.062574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.066684] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4577][T1002_TimerWK0_Flush][T1002][Y0-0000000000000000-0-0] [lt=13] ====== traversal_flush timer task ====== [2025-05-15 09:40:41.066749] INFO [MDS] flush (mds_table_mgr.cpp:161) [4577][T1002_TimerWK0_Flush][T1002][YB427F000001-000635291FCC6696-0-0] [lt=17] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1a6ea99150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:41.066876] INFO [MDS] flush (mds_table_mgr.cpp:161) [4577][T1002_TimerWK0_Flush][T1002][YB427F000001-000635291FCC6696-0-0] [lt=99] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1001}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1a6eb09150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:41.066941] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4577][T1002_TimerWK0_Flush][T1002][YB427F000001-000635291FCC6696-0-0] [lt=63] succeed to traversal_flush(ret=0, ls_cnt=2) [2025-05-15 09:40:41.070443] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=11] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:41.070473] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.070587] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=114][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:41.070598] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:41.070608] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:41.070662] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=54][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:41.072149] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.072190] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.072196] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC96B6-0-0] [lt=49][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302041072171}) [2025-05-15 09:40:41.072217] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:41.072234] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:41.072248] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.072260] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts statistics(tenant_id=1001, gts_rpc_cnt=0, get_gts_cache_cnt=9107, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:41.072273] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.072643] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:41.072683] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:41.072684] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.072698] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.072708] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.082791] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.087777] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=39] new mysql sessid created(conn.sessid_=3221553221, support_ssl=false) [2025-05-15 09:40:41.087903] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553221, sess.client_addr_="172.22.0.1:40076") [2025-05-15 09:40:41.087924] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553221, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.088043] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.088080] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.088109] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.088145] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=34][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.088164] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=18][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.088171] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.088198] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.088217] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553221, proxy_sessid=0) [2025-05-15 09:40:41.088249] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=15][errcode=0] server close connection(sessid=3221553221, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.088264] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553221, proxy_sessid=0) [2025-05-15 09:40:41.088272] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.088286] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9095-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553221, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.088344] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553221, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041087579, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.088368] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553221, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.088387] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553221, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041087579, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.088409] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553221) [2025-05-15 09:40:41.088430] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553221, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.092903] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.102853] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4318][T1_LockWaitMgr][T1][Y0-0000000000000000-0-0] [lt=22] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:41.103016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.106644] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.107568] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.107597] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041107560) [2025-05-15 09:40:41.107618] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302040907551, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.107657] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] get wrs ts(ls_id={id:1}, delta=12343356423, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.107677] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.107701] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.107720] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.107735] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.107757] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1001}, delta=12343356423, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.107773] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.107795] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.107808] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.107816] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.107830] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.107839] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.107848] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041107636) [2025-05-15 09:40:41.110437] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.110458] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3161259, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.110475] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3161259, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.110500] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.110517] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3161259, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.110533] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.110550] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.110563] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.110578] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.110586] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.110602] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.110618] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041110617, replica_locations:[]}) [2025-05-15 09:40:41.110636] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.110661] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.110672] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:41.110716] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.110764] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11805001240, cache_obj->added_lc()=false, cache_obj->get_object_id()=15835, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.111833] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.111853] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.111868] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.111882] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.111890] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.111904] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.111919] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.111930] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.111943] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.111955] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.111969] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.111983] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041111982, replica_locations:[]}) [2025-05-15 09:40:41.112046] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=3159635, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=313, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.113125] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.115954] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:41.115972] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:41.115981] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=8] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:41.115998] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:41.116124] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=23] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:41.116156] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=17] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:41.119566] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.119588] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.119611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.119627] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.119640] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.119651] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.119662] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.119675] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.119685] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302041119537, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.119698] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.119708] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.119776] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:41.119787] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:41.119795] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:41.119804] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:41.120477] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=18] gc stale ls task succ [2025-05-15 09:40:41.123238] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.129027] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:41.129055] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-05-15 09:40:41.129067] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=11] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:41.129084] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:41.129100] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11804019449, flush_io_finished_ret_:0, flush_io_finished_round_:1515, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:41.129232] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.129252] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041129226) [2025-05-15 09:40:41.129266] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302040929215, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.129289] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12343400903, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.129304] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12343228410, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.129306] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.129323] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.129327] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.129338] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.129348] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.129351] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.129360] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.129365] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.129367] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.129374] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.129374] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041129278) [2025-05-15 09:40:41.129387] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.129396] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.129405] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041129290) [2025-05-15 09:40:41.131778] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553223, support_ssl=false) [2025-05-15 09:40:41.131919] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553223, sess.client_addr_="172.22.0.1:40078") [2025-05-15 09:40:41.131937] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553223, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.132081] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553224, support_ssl=false) [2025-05-15 09:40:41.132117] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.132158] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.132166] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553224, sess.client_addr_="172.22.0.1:40088") [2025-05-15 09:40:41.132178] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221553225, support_ssl=false) [2025-05-15 09:40:41.132188] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=28][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.132213] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553225, sess.client_addr_="172.22.0.1:40092") [2025-05-15 09:40:41.132211] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.132203] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=33] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553224, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.132227] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.132226] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553225, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.132236] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.132257] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.132276] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553223, proxy_sessid=0) [2025-05-15 09:40:41.132309] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=15][errcode=0] server close connection(sessid=3221553223, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.132325] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553223, proxy_sessid=0) [2025-05-15 09:40:41.132333] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.132346] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A6-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553223, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.132388] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.132394] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.132415] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.132427] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.132437] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.132451] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553223, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041131906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.132459] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.132455] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.132472] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.132474] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553223, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.132483] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.132481] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=26][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.132486] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553223, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041131906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.132511] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=29][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.132517] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] mark session id unused(conn.sessid_=3221553223) [2025-05-15 09:40:41.132519] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.132518] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=27] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.132534] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553225, proxy_sessid=0) [2025-05-15 09:40:41.132541] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.132558] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553224, proxy_sessid=0) [2025-05-15 09:40:41.132541] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553223, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.132563] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=14][errcode=0] server close connection(sessid=3221553225, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.132574] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553225, proxy_sessid=0) [2025-05-15 09:40:41.132582] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=9][errcode=0] server close connection(sessid=3221553224, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.132587] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:41.132594] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553224, proxy_sessid=0) [2025-05-15 09:40:41.132596] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D6-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553225, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.132609] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=14][errcode=-4016] session is null [2025-05-15 09:40:41.132617] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9096-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553224, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.132643] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553225, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302041131906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.132664] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553225, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.132674] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553224, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041131906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.132699] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553224, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.132703] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553225, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302041131906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.132719] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553225) [2025-05-15 09:40:41.132711] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553224, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041131906, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.132728] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553224) [2025-05-15 09:40:41.132733] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553225, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.132746] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553224, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.133354] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.133360] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221553226, support_ssl=false) [2025-05-15 09:40:41.133402] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553226, sess.client_addr_="172.22.0.1:40108") [2025-05-15 09:40:41.133419] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553226, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.133686] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.133712] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.133731] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.133753] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.133761] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.133760] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=31] new mysql sessid created(conn.sessid_=3221553227, support_ssl=false) [2025-05-15 09:40:41.133773] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.133787] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.133801] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553226, proxy_sessid=0) [2025-05-15 09:40:41.133812] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221553227, sess.client_addr_="172.22.0.1:40116") [2025-05-15 09:40:41.133825] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=12][errcode=0] server close connection(sessid=3221553226, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.133837] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553226, proxy_sessid=0) [2025-05-15 09:40:41.133845] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:41.133838] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553227, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.133857] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A7-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553226, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.133903] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553226, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302041132964, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.133925] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553226, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.133935] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553226, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302041132964, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.133952] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553226) [2025-05-15 09:40:41.133963] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553226, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.134006] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.134033] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.134048] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.134063] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.134071] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=6][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.134081] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.134092] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=6] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.134105] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553227, proxy_sessid=0) [2025-05-15 09:40:41.134124] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=8][errcode=0] server close connection(sessid=3221553227, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.134135] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553227, proxy_sessid=0) [2025-05-15 09:40:41.134142] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.134153] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D7-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553227, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.134198] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553227, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041132964, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.134221] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553227, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.134235] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553227, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041132964, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.134252] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553227) [2025-05-15 09:40:41.134263] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] new mysql sessid created(conn.sessid_=3221553228, support_ssl=false) [2025-05-15 09:40:41.134264] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553227, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.134299] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553228, sess.client_addr_="172.22.0.1:40128") [2025-05-15 09:40:41.134314] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553228, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.134554] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.134582] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.134600] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.134620] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.134629] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.134636] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.134655] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.134670] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553228, proxy_sessid=0) [2025-05-15 09:40:41.134686] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] new mysql sessid created(conn.sessid_=3221553229, support_ssl=false) [2025-05-15 09:40:41.134694] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=13][errcode=0] server close connection(sessid=3221553228, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.134706] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553228, proxy_sessid=0) [2025-05-15 09:40:41.134715] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.134734] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553229, sess.client_addr_="172.22.0.1:40132") [2025-05-15 09:40:41.134728] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9097-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553228, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.134750] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de14030, session_id:3221553229, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.134785] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553228, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041134023, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.134805] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553228, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.134815] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553228, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041134023, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.134831] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553228) [2025-05-15 09:40:41.134844] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553228, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.134875] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] new mysql sessid created(conn.sessid_=3221553230, support_ssl=false) [2025-05-15 09:40:41.134909] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] sm conn init succ(conn.sessid_=3221553230, sess.client_addr_="172.22.0.1:40140") [2025-05-15 09:40:41.134922] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de14db0, session_id:3221553230, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.134996] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.135019] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.135041] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.135072] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.135068] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=26][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.135089] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.135100] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.135106] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=17][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.135121] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.135130] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135140] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.135145] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553229, proxy_sessid=0) [2025-05-15 09:40:41.135148] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.135158] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.135168] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=11][errcode=0] server close connection(sessid=3221553229, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135177] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553229, proxy_sessid=0) [2025-05-15 09:40:41.135178] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135182] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] new mysql sessid created(conn.sessid_=3221553231, support_ssl=false) [2025-05-15 09:40:41.135190] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553230, proxy_sessid=0) [2025-05-15 09:40:41.135189] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:41.135208] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=7][errcode=0] server close connection(sessid=3221553230, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135199] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A8-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553229, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.135220] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553230, proxy_sessid=0) [2025-05-15 09:40:41.135221] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553231, sess.client_addr_="172.22.0.1:40156") [2025-05-15 09:40:41.135227] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.135234] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553231, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.135234] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D8-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553230, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.135268] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14030, session_id:3221553229, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302041134023, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.135299] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de14db0, session_id:3221553230, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041134023, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.135322] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553229, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.135336] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] kill and revert session(conn.sessid_=3221553230, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.135342] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a1de14030, session_id:3221553229, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302041134023, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.135359] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553229) [2025-05-15 09:40:41.135370] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553229, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.135389] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a1de14db0, session_id:3221553230, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041134023, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.135404] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553230) [2025-05-15 09:40:41.135413] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553230, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.135426] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221553232, support_ssl=false) [2025-05-15 09:40:41.135433] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553233, support_ssl=false) [2025-05-15 09:40:41.135471] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221553232, sess.client_addr_="172.22.0.1:40170") [2025-05-15 09:40:41.135480] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553233, sess.client_addr_="172.22.0.1:40166") [2025-05-15 09:40:41.135486] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553232, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.135503] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.135509] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553233, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.135526] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.135545] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.135564] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.135573] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.135587] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.135600] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135616] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553231, proxy_sessid=0) [2025-05-15 09:40:41.135634] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=8][errcode=0] server close connection(sessid=3221553231, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135647] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553231, proxy_sessid=0) [2025-05-15 09:40:41.135654] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.135672] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.135666] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9098-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553231, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.135701] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.135717] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553231, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041135081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.135720] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.135739] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553231, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.135740] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=20][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.135751] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.135751] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553231, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041135081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.135758] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.135769] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553231) [2025-05-15 09:40:41.135777] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135738] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.135782] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553231, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.135793] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553232, proxy_sessid=0) [2025-05-15 09:40:41.135814] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=9][errcode=0] server close connection(sessid=3221553232, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135826] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553232, proxy_sessid=0) [2025-05-15 09:40:41.135833] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.135829] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=54] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.135840] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90A9-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553232, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.135866] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.135889] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553232, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302041135081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.135898] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=30][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.135914] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] kill and revert session(conn.sessid_=3221553232, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.135916] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.135933] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=17][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.135932] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553232, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302041135081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.135952] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553232) [2025-05-15 09:40:41.135965] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553232, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.135971] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.135994] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221553233, proxy_sessid=0) [2025-05-15 09:40:41.136036] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=17][errcode=0] server close connection(sessid=3221553233, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.136052] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553233, proxy_sessid=0) [2025-05-15 09:40:41.136062] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:41.136077] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90D9-0-0] [lt=13] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553233, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.136140] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553233, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302041135081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.136176] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] kill and revert session(conn.sessid_=3221553233, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.136195] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553233, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302041135081, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.136216] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553233) [2025-05-15 09:40:41.136246] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553233, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.143465] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.150816] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=28] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:41.153594] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.163704] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.173234] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.173811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.173950] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.174060] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.174424] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.183920] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.186711] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=19] ====== check clog disk timer task ====== [2025-05-15 09:40:41.186732] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4291][T1_TimerWK2_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=17] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:41.192085] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=19] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:41.193988] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.200848] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=17] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:41.200878] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=27][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:41.200894] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=15][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:41.200910] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=15][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:41.200923] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=9][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:41.200934] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:41.200947] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=8][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:41.200959] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:41.200972] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=12][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:41.200983] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:41.200993] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:41.201004] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:41.201016] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:41.201027] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:41.201050] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:41.201063] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.201076] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.201089] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:41.201102] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:41.201119] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:41.201126] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=7][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.201148] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:41.201170] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.201182] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.201189] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:41.201209] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:41.201222] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4463][T1001_TimerWK3_LSMetaCh][T1][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.201236] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:41.201250] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=14][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.201263] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:41.201274] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:41.201286] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca3b0, start=1747302041200566, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.201300] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:41.201310] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.201408] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=8][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:41.201423] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:41.201438] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=13][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:41.201450] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:41.201464] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=11][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:41.201478] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=12][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:41.201501] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4463][T1001_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F5C6472-0-0] [lt=23][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:41.204100] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.206710] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.207647] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12343455650, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.207674] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.207699] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.207718] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.207728] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.207743] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12343455650, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.207756] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.207771] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.207784] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.207792] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.207805] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.207816] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.207827] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041207624) [2025-05-15 09:40:41.212192] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.212215] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3059512, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.212236] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=3059512, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.212255] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.212271] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=3059512, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.212288] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.212311] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.212331] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.212350] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.212365] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.212385] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.212406] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041212403, replica_locations:[]}) [2025-05-15 09:40:41.212426] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.212455] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.212468] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:41.212514] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.212573] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11805103050, cache_obj->added_lc()=false, cache_obj->get_object_id()=15836, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.213899] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.213924] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.213939] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.213953] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.213967] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.213985] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.214004] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.214021] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.214038] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.214054] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.214073] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.214089] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041214088, replica_locations:[]}) [2025-05-15 09:40:41.214161] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1] will sleep(sleep_us=100000, remain_us=3057521, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=314, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.214208] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.219481] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.219510] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=30][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.219525] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.219541] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.219555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.219570] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.219582] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.219593] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.219605] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302041219463, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.219617] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.219629] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.219682] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:41.219696] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:41.219711] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:41.219722] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:41.224328] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.229291] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.229330] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041229284) [2025-05-15 09:40:41.229335] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.229350] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302041129277, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.229364] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041229331) [2025-05-15 09:40:41.229379] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302041029290, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.229376] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12343501152, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.229403] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12343328659, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.229395] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.229417] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.229418] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.229432] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.229433] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.229440] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.229447] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.229452] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.229457] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.229460] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.229467] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.229468] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041229364) [2025-05-15 09:40:41.229478] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.229484] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041229391) [2025-05-15 09:40:41.234454] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.244578] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.247313] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-05-15 09:40:41.249001] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4462][T1001_TimerWK2_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=19] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:41.254687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.256549] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.256565] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:41.256574] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:41.256590] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:41.256604] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=10398720) [2025-05-15 09:40:41.256629] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=25, size_used=0, mem_used=0) [2025-05-15 09:40:41.256656] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4460][T1001_TimerWK0_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:41.259602] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:41.259629] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=20] temporary file module use no blocks [2025-05-15 09:40:41.259637] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=7] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:41.259655] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:41.259673] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11804150020, flush_io_finished_ret_:0, flush_io_finished_round_:1515, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:41.264796] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.268942] INFO [MDS] for_each_ls_in_tenant (mds_tenant_service.cpp:225) [4416][T1001_Occam][T1001][YB427F000001-000635291CEC65D1-0-0] [lt=22] for each ls(succ_num=1, ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.270585] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=16] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:41.274901] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.285377] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.296048] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.306161] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.306768] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.307687] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.307715] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041307680) [2025-05-15 09:40:41.307730] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302041107635, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.307761] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12343555865, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.307779] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.307805] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.307829] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.307843] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.307861] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12343555865, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.307874] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.307880] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=15][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:41.307889] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.307895] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.307900] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.307907] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.307917] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.307925] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.307913] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.307937] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041307743) [2025-05-15 09:40:41.307935] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.307950] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.307964] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.307981] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.308000] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.308020] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.308037] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.308049] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.308067] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.308098] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=13] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.308114] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=13] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:41.308477] WDIAG load_file_to_string (utility.h:673) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:41.308505] WDIAG get_ethernet_speed (utility.cpp:575) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:41.308519] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:41.308531] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:41.309896] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=14] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:41.309925] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=25][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:41.309940] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:41.309955] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:41.309979] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=20][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:41.309993] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=14][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:41.310008] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:41.310020] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:41.310031] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:41.310059] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=26][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:41.310067] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=7][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:41.310080] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=12][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:41.310089] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:41.310100] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:41.310124] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=12][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:41.310136] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.310150] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.310163] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:41.310175] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:41.310190] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:41.310201] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.310222] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=16][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:41.310245] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.310255] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.310266] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:41.310281] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=6][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:41.310295] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.310310] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:41.310323] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.310335] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:41.310343] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=7][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:41.310354] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb440, start=1747302041309645, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.310368] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=14][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:41.310380] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.310471] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:41.310486] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=14][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:41.310509] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=23][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:41.310521] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:41.310533] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:41.310545] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=11][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:41.310553] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4289][T1_TimerWK0_LSMetaCh][T1][YB427F000001-000635291C0C63C3-0-0] [lt=8][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:41.314283] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.314306] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2957413, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.314327] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2957413, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.314346] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.314360] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2957413, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.314377] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.314396] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.314413] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.314429] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.314444] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.314464] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.314482] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041314480, replica_locations:[]}) [2025-05-15 09:40:41.314512] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.314543] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.314556] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:41.314600] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.314653] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11805205127, cache_obj->added_lc()=false, cache_obj->get_object_id()=15837, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.315784] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.315808] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.315825] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.315844] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.315858] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.315874] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.315893] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.315910] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.315931] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.315947] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.315966] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.315983] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041315982, replica_locations:[]}) [2025-05-15 09:40:41.316051] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2955631, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=315, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.316271] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.319515] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.319536] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.319552] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.319567] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.319582] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.319593] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.319604] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.319612] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.319620] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302041319502, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.319632] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.319643] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.319687] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:41.319700] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:41.319712] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:41.319724] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:41.326386] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.329358] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12343601314, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.329382] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.329405] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23] get wrs ts(ls_id={id:1}, delta=12343428821, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.329405] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.329422] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.329423] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.329439] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.329450] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.329436] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.329458] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.329465] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=24][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.329472] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.329474] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.329481] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.329483] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041329342) [2025-05-15 09:40:41.329502] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041329392) [2025-05-15 09:40:41.330727] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4289][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=7] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:41.330730] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4291][T1_TimerWK2_Flush][T1][Y0-0000000000000000-0-0] [lt=13] ====== traversal_flush timer task ====== [2025-05-15 09:40:41.330735] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=16] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:4, threads_idle_time:4845269607, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:41.330746] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4289][T1_TimerWK0_TabletShell][T1][Y0-0000000000000000-0-0] [lt=14] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=85) [2025-05-15 09:40:41.330800] INFO [MDS] flush (mds_table_mgr.cpp:161) [4291][T1_TimerWK2_Flush][T1][YB427F000001-000635291D0C63C1-0-0] [lt=14] [FREEZE]no valid mds table there, no need do flush(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:0, ordered_tablets:{}, max_consequent_callbacked_scn={val:18446744073709551615, v:3}, *this={this:0x7f1aaf36f150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:41.330902] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4291][T1_TimerWK2_Flush][T1][YB427F000001-000635291D0C63C1-0-0] [lt=97] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-15 09:40:41.331000] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [4292][T1_TimerWK3][T1][Y0-0000000000000000-0-0] [lt=23] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:2, threads_idle_time:4845269607, tenant_id:1}, cur_thread_count=4, cnt=1, new_thread_count=5) [2025-05-15 09:40:41.336503] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.341090] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4289][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=12] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:41.341107] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4289][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=14] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:41.341121] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4289][T1_TimerWK0_][T1][Y0-0000000000000000-0-0] [lt=12] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=88) [2025-05-15 09:40:41.346615] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.351117] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:41.355783] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:41.355813] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.355827] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:41.355838] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:41.355848] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:41.355916] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=68][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:41.355931] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.355941] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:41.355948] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:41.355954] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:41.356007] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=53][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:41.356723] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.366834] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.368224] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:41.368292] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:41.368360] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:41.368421] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:41.368450] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=13] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:18271, total:18282) [2025-05-15 09:40:41.368485] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:41.368546] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=56] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:18271, total:18282) [2025-05-15 09:40:41.368577] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:41.368608] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:41.368625] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:41.368650] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] successfully alloc block(macro_id={[ver=1,mode=0,seq=628776][2nd=1903]}) [2025-05-15 09:40:41.376977] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.387018] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:362, tid:4290}]) [2025-05-15 09:40:41.387084] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.389275] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=13] table not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, table_name.ptr()="data_size:27, data:5F5F616C6C5F7669727475616C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:41.389309] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=31][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_virtual_ls_meta_table, ret=-5019) [2025-05-15 09:40:41.389323] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_virtual_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:41.389337] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=13][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_virtual_ls_meta_table) [2025-05-15 09:40:41.389351] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:41.389362] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=10][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:41.389377] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] Table 'oceanbase.__all_virtual_ls_meta_table' doesn't exist [2025-05-15 09:40:41.389387] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:41.389397] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:41.389407] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:41.389416] WDIAG [SQL.RESV] resolve_joined_table_item (ob_dml_resolver.cpp:6094) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] resolve table failed(ret=-5019) [2025-05-15 09:40:41.389427] WDIAG [SQL.RESV] resolve_joined_table (ob_dml_resolver.cpp:5649) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] resolve joined table item failed(ret=-5019) [2025-05-15 09:40:41.389437] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5457) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] resolve joined table failed(ret=-5019) [2025-05-15 09:40:41.389445] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:41.389452] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=6][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:41.389461] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:41.389470] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:41.389485] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=8][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:41.389506] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=20][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.389518] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.389528] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:41.389537] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;, ret=-5019) [2025-05-15 09:40:41.389549] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=10][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:41.389557] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.389575] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:41.389594] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.389603] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=8][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.389611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:41.389627] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:41.389639] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4776][BlackListServic][T1][YB427F000001-000635291F1C630B-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.389650] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4776][BlackListServic][T0][YB427F000001-000635291F1C630B-0-0] [lt=9][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;"}, aret=-5019, ret=-5019) [2025-05-15 09:40:41.389661] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:41.389673] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:41.389683] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:41.389699] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302041388940, sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:41.389711] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:114) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:41.389722] WDIAG [STORAGE.TRANS] do_thread_task_ (ob_black_list.cpp:219) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] fail to execute sql(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=select a.svr_ip, a.svr_port, a.tenant_id, a.ls_id, a.role, nvl(b.weak_read_scn, 1) as weak_read_scn, nvl(b.migrate_status, 0) as migrate_status, nvl(b.tx_blocked, 0) as tx_blocked from oceanbase.__all_virtual_ls_meta_table a left join oceanbase.__all_virtual_ls_info b on a.svr_ip = b.svr_ip and a.svr_port = b.svr_port and a.tenant_id = b.tenant_id and a.ls_id = b.ls_id;) [2025-05-15 09:40:41.389804] INFO [STORAGE.TRANS] run1 (ob_black_list.cpp:191) [4776][BlackListServic][T0][Y0-0000000000000000-0-0] [lt=12] ls blacklist refresh finish(cost_time=2294) [2025-05-15 09:40:41.395327] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.395335] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.395350] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=16] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.395435] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=15] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.395741] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.395943] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=31] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.396215] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=41] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:41.397198] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.397858] INFO [CLOG] run1 (ob_garbage_collector.cpp:1449) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13] Garbage Collector is running(seq_=152, gc_interval=10000000) [2025-05-15 09:40:41.397894] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:41.397912] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.397944] INFO [CLOG] gc_check_member_list_ (ob_garbage_collector.cpp:1542) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=15] gc_check_member_list_ cost time(ret=0, time_us=65) [2025-05-15 09:40:41.397960] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] execute_gc cost time(ret=0, time_us=1) [2025-05-15 09:40:41.399610] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=10] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, table_name.ptr()="data_size:15, data:5F5F616C6C5F6C735F737461747573", ret=-5019) [2025-05-15 09:40:41.399637] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=24][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_status, ret=-5019) [2025-05-15 09:40:41.399652] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=15][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_status, db_name=oceanbase) [2025-05-15 09:40:41.399666] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_status) [2025-05-15 09:40:41.399688] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=18][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:41.399699] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:41.399710] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_ls_status' doesn't exist [2025-05-15 09:40:41.399721] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:41.399732] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:41.399742] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=9][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:41.399750] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:41.399761] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=11][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:41.399773] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=11][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:41.399784] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:41.399804] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:41.399816] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.399829] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=11][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.399841] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:41.399851] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001, ret=-5019) [2025-05-15 09:40:41.399865] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:41.399876] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=9][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.399896] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=15][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:41.399916] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.399924] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.399933] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:41.399948] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=6][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:41.399962] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4482][T1001_GCCollect][T1][YB427F000001-000635291FEC61A9-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.399973] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4482][T1001_GCCollect][T1001][YB427F000001-000635291FEC61A9-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001"}, aret=-5019, ret=-5019) [2025-05-15 09:40:41.399987] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-05-15 09:40:41.399998] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:41.400010] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:41.400021] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302041399378, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-05-15 09:40:41.400033] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:41.400042] WDIAG [SHARE] inner_get_ls_status_ (ob_ls_status_operator.cpp:980) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to read(ret=-5019, ret="OB_TABLE_NOT_EXIST", exec_tenant_id=1, sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001) [2025-05-15 09:40:41.400130] WDIAG [SHARE] get_ls_status_ (ob_ls_status_operator.cpp:1062) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] fail to inner get ls status info(ret=-5019, ret="OB_TABLE_NOT_EXIST", sql=SELECT * FROM __all_ls_status where ls_id = 1 and tenant_id = 1001, tenant_id=1001, exec_tenant_id=1, need_member_list=false) [2025-05-15 09:40:41.400145] WDIAG [SHARE] get_ls_status_info (ob_ls_status_operator.cpp:626) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] failed to get ls status(ret=-5019, ret="OB_TABLE_NOT_EXIST", id={id:1}, tenant_id=1001) [2025-05-15 09:40:41.400157] INFO [CLOG] get_ls_status_from_table (ob_garbage_collector.cpp:1481) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=11] failed to get ls status info from table(ret=-5019, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:41.400168] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1766) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] failed to get ls status from table(ret=-5019, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:41.400181] WDIAG [CLOG] gc_check_ls_status_ (ob_garbage_collector.cpp:1666) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] get_ls_status_ failed(tmp_ret=-5019, ls->get_ls_id()={id:1}) [2025-05-15 09:40:41.400194] INFO [CLOG] execute_gc_ (ob_garbage_collector.cpp:1854) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] execute_gc cost time(ret=0, time_us=1) [2025-05-15 09:40:41.400204] INFO [SERVER] handle (ob_safe_destroy_handler.cpp:240) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=9] ObSafeDestroyHandler start process [2025-05-15 09:40:41.400212] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:133) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=7] ObSafeDestroyTaskQueue::loop begin(queue_.size()=0) [2025-05-15 09:40:41.400229] INFO [SERVER] loop (ob_safe_destroy_handler.cpp:140) [4482][T1001_GCCollect][T1001][Y0-0000000000000000-0-0] [lt=10] ObSafeDestroyTaskQueue::loop finish(ret=0, queue_.size()=0) [2025-05-15 09:40:41.406827] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.407304] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.407766] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12343656130, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.407791] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.407816] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.407835] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.407848] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.407864] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12343656130, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.407877] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.407892] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.407902] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.407913] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.407922] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.407949] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.407960] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041407744) [2025-05-15 09:40:41.414849] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:41.414868] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.414879] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:41.415002] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=116][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:41.416187] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.416210] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2855513, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.416227] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2855513, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.416242] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.416252] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2855513, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.416268] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.416284] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.416298] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.416312] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.416324] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.416340] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.416357] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041416355, replica_locations:[]}) [2025-05-15 09:40:41.416376] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.416393] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:315, local_retry_times:315, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:41.416412] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.416423] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:41.416441] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:41.416452] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:41.416460] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:41.416472] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:41.416483] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.416534] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11805307012, cache_obj->added_lc()=false, cache_obj->get_object_id()=15838, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.417414] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.417661] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.417689] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.417721] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.417737] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.417751] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.417763] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.417781] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.417793] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.417807] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.417818] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.417830] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.417840] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.417854] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.417868] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041417866, replica_locations:[]}) [2025-05-15 09:40:41.417883] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.417896] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.417908] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.417922] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:41.417932] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:41.417943] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:41.417960] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:41.417974] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:41.417983] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:41.417994] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:41.418002] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:41.418012] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:41.418023] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:41.418034] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:41.418043] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:41.418052] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:41.418062] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:41.418071] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:41.418081] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:41.418096] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:41.418105] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.418115] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:41.418124] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:41.418133] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:41.418142] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=316, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.418166] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] will sleep(sleep_us=100000, remain_us=2853516, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=316, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.419626] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.419646] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.419661] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.419677] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.419692] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.419704] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.419716] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.419727] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.419739] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302041419610, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.419753] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.419761] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.419811] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:41.419823] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:41.419835] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:41.419845] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:41.424782] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.424803] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.424819] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:41.426067] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:41.426089] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.426105] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:41.426121] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:41.426131] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:41.426142] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:41.427527] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.429406] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:41.429425] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.429449] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041429401) [2025-05-15 09:40:41.429450] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.429465] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302041229363, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.429468] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041429446) [2025-05-15 09:40:41.429481] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302041229391, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.429498] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12343700634, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.429517] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25] get wrs ts(ls_id={id:1}, delta=12343528141, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.429515] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.429531] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.429534] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.429548] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.429549] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.429556] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.429562] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.429567] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.429573] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.429576] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.429584] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.429585] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041429478) [2025-05-15 09:40:41.429594] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.429600] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041429506) [2025-05-15 09:40:41.436680] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:41.436699] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.436709] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:41.437642] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.442121] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] ====== checkpoint timer task ====== [2025-05-15 09:40:41.442767] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] succ to get rec scn(*this={this:0x7f1ab0204030, ls_id:{id:1}, tenant_id:1, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300517859523, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1303}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.442806] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=39] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a96ba2080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a96ba2080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.442867] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=31] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1aaf36f150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274404592930000, v:0}, clog_base_lsn:{lsn:2818400256}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747286979635229003, v:0}, all_id_meta:{id_meta:[{limited_id:1747289709556135000, latest_log_ts:{val:1747289692924660001, v:0}}, {limited_id:8000001, latest_log_ts:{val:1747229536489984002, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186180030177001, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1ac01fa030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:41.442957] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=88] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1aaf36e1b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1aaf379010, memtable_head:0, memtable_tail:1, t3m:0x7f1ac66cc030, tables:[0x7f1a96b92080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1aaf37ad10, ls_tablet_svr:0x7f1aaf36e190}, min_rec_scn={val:1747274404592930000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:41.442999] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=35] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-15 09:40:41.443016] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=0, rec_log_ts={val:1747289692924660001, v:0}) [2025-05-15 09:40:41.443029] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.443037] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.443053] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:1747277617103769000, v:0}, flushed_scn_={val:1746499972147257000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:1747277617937058000, v:0}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:41.443074] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] rec_scn of ObLockMemtable is (rec_scn_={val:1747277617103769000, v:0}, flushed_scn_={val:1746499972147257000, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:1747277617937058000, v:0}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:41.443095] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=17] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274404592930000, v:0}, checkpoint_scn_in_ls_meta={val:1747274404592930000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:41.443118] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=14][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:2818400256}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:2818400256}, lsn={lsn:2818400256}) [2025-05-15 09:40:41.443143] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:2818400256}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:41.443153] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4292][T1_TimerWK3_TxCkpt][T1][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-15 09:40:41.447417] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=31] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=0) [2025-05-15 09:40:41.447751] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.456205] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:133) [3956][TimerWK3_ServerTracerTim][T0][YB427F000001-00063529204C618B-0-0] [lt=11][errcode=-4002] invalid argument, empty server list(ret=-4002) [2025-05-15 09:40:41.456222] WDIAG [SHARE] refresh (ob_alive_server_tracer.cpp:378) [3956][TimerWK3_ServerTracerTim][T0][YB427F000001-00063529204C618B-0-0] [lt=17][errcode=-4002] refresh sever list failed(ret=-4002) [2025-05-15 09:40:41.456229] WDIAG [SHARE] runTimerTask (ob_alive_server_tracer.cpp:252) [3956][TimerWK3_ServerTracerTim][T0][YB427F000001-00063529204C618B-0-0] [lt=7][errcode=-4002] refresh alive server list failed(ret=-4002) [2025-05-15 09:40:41.457857] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.468975] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.473389] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.473410] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.473425] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:41.473448] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4292][T1_TimerWK3_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:41.479090] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.489202] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.499313] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.506892] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.507212] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:41.507242] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.507261] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:41.507288] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:41.507302] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:41.507315] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:41.507327] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:41.507346] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4579][T1002_TimerWK2_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:41.507518] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:41.507539] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:41.507548] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:41.507560] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:41.507572] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:41.507586] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:41.507602] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:41.507613] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:41.507655] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:41.507804] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.507837] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041507797) [2025-05-15 09:40:41.507852] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302041307742, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.507883] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12343756613, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.507901] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.507922] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.507938] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.507949] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.507965] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12343756613, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.507978] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.507993] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.508005] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.508015] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.508028] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.508039] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.508045] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041507866) [2025-05-15 09:40:41.509423] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.512573] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=11] ====== checkpoint timer task ====== [2025-05-15 09:40:41.513226] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=17] succ to get rec scn(*this={this:0x7f1a69e04030, ls_id:{id:1}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300524716821, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:726}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.513267] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=40] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a690fa080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a690fa080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.513328] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=33] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1a6ea99150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:41.513414] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=82] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1a6ea981b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1a6eaa3010, memtable_head:0, memtable_tail:1, t3m:0x7f1a8f8ea030, tables:[0x7f1a690ea080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1a6eaa4d10, ls_tablet_svr:0x7f1a6ea98190}, min_rec_scn={val:1747274406063927000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:41.513449] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=28] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1747274406063927000, v:0}) [2025-05-15 09:40:41.513469] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=17] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.513482] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=14] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.513510] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=18] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:41.513535] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=20] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:41.513553] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274406063927000, v:0}, checkpoint_scn_in_ls_meta={val:1747274406063927000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:41.513577] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:872361984}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:872361984}, lsn={lsn:872361984}) [2025-05-15 09:40:41.513602] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=25] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:872361984}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:41.514195] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=8] succ to get rec scn(*this={this:0x7f1a69e76030, ls_id:{id:1001}, tenant_id:1002, tx_ls_state_mgr:{ls_id_:{id:1001}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300524741888, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:5}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.514222] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=26] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a690fb390, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a690fb390, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1001}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:41.514262] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:249) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=21] [FREEZE]no valid mds table there, return MAX SCN(ret=0, ret="OB_SUCCESS", min_rec_scn={val:4611686018427387903, v:0}, order_flusher_for_one={high_priority_flusher:size:0, ordered_tablets:{}, *this={this:0x7f1a6eb09150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:41.514321] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=58] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="ALL_EMPTY", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]=NULL, min_rec_scn={val:4611686018427387903, v:0}, ls_id_={id:1001}) [2025-05-15 09:40:41.514330] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=8] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1001}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1747274672721723000, v:0}) [2025-05-15 09:40:41.514344] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=13] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-05-15 09:40:41.514361] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=16] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1001}) [2025-05-15 09:40:41.514377] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=16] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747289697751194001, v:0}, checkpoint_scn_in_ls_meta={val:1747289697751194001, v:0}, ls_id={id:1001}, service_type="MAX_DECIDED_SCN") [2025-05-15 09:40:41.514388] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1001, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:738152448}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1746497901775524000, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:738152448}, lsn={lsn:738152448}) [2025-05-15 09:40:41.514407] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=19] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:738152448}, ls->get_ls_id()={id:1001}) [2025-05-15 09:40:41.514415] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4579][T1002_TimerWK2_TxCkpt][T1002][Y0-0000000000000000-0-0] [lt=7] succeed to update_clog_checkpoint(ret=0, ls_cnt=2) [2025-05-15 09:40:41.518289] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.518315] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2753409, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.518331] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2753409, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.518345] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.518355] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2753409, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.518367] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.518382] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.518395] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.518409] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.518420] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.518434] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.518450] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041518448, replica_locations:[]}) [2025-05-15 09:40:41.518467] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.518484] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:316, local_retry_times:316, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:41.518514] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.518526] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:41.518541] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:41.518550] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:41.518559] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:41.518592] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:41.518607] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.518655] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11805409131, cache_obj->added_lc()=false, cache_obj->get_object_id()=15839, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.519565] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.519756] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.519775] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.519778] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.519795] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.519806] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.519809] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.519821] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.519836] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.519839] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.519848] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.519852] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.519860] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.519864] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.519872] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302041519735, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.519875] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.519887] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.519888] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.519901] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.519901] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.519913] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.519927] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.519940] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.519950] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.519963] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:41.519963] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.519974] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:41.519977] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041519976, replica_locations:[]}) [2025-05-15 09:40:41.519987] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:41.519993] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.519997] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:41.520006] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.520019] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.520034] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:41.520043] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:41.520053] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:41.520070] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:41.520083] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:41.520093] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:41.520102] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:41.520111] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:41.520120] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:41.520132] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:41.520144] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:41.520153] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:41.520160] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=5][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:41.520169] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:41.520178] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:41.520188] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:41.520203] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:41.520213] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.520223] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:41.520233] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:41.520244] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:41.520252] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=317, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.520275] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] will sleep(sleep_us=100000, remain_us=2751406, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=317, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.524839] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.524861] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.524872] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:41.526124] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:41.526148] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.526160] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:41.526174] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:41.526183] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:41.526194] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:41.529479] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12343801089, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.529515] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.529518] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12343628596, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.529536] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.529536] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.529551] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.529552] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.529565] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.529564] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.529572] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.529577] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.529585] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.529589] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.529594] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.529596] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041529465) [2025-05-15 09:40:41.529601] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041529505) [2025-05-15 09:40:41.529691] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.536784] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:41.536803] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.536813] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:41.539797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.542501] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=18] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:41.544994] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=20] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:41.549904] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.551386] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:41.560012] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.570124] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.573124] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.573146] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.573171] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:41.573185] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:41.573204] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.573220] INFO [STORAGE.TRANS] refresh_gts (ob_gts_source.cpp:509) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15] refresh gts(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1, need_refresh=false, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.573233] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] refresh gts failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:41.573246] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:41.573339] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=93][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:41.573347] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.573368] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:41.573379] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:41.573392] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:41.573404] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:41.573411] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:41.573421] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:41.573431] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.573780] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:41.573793] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:41.573807] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:41.573817] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:41.573824] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:41.573834] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:41.573845] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:41.573853] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:41.573868] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:41.580234] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.581048] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6608-0-0] [lt=15][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.581068] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6608-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.581093] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6608-0-0] [lt=13][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.581109] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4580][T1002_TimerWK3_LSMetaCh][T1001][YB427F000001-000635291F7C6608-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.581121] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.581134] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.581148] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.581160] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.581173] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302041581024, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.581187] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.581196] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=7][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:41.581268] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=12][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:41.581281] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=12][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:41.581294] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=12][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:41.581305] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=10][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:41.581315] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=7][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:41.581328] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=11][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:41.581339] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4580][T1002_TimerWK3_LSMetaCh][T1002][YB427F000001-000635291F7C6608-0-0] [lt=10][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:41.589001] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221553242, support_ssl=false) [2025-05-15 09:40:41.589081] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=35] sm conn init succ(conn.sessid_=3221553242, sess.client_addr_="172.22.0.1:40184") [2025-05-15 09:40:41.589102] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553242, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.589334] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.589372] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.589403] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.589440] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=35][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.589457] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.589466] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.589503] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.589522] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553242, proxy_sessid=0) [2025-05-15 09:40:41.589554] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=15][errcode=0] server close connection(sessid=3221553242, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.589568] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553242, proxy_sessid=0) [2025-05-15 09:40:41.589576] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.589584] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC9099-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553242, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.589641] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553242, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302041589139, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.589668] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=25] kill and revert session(conn.sessid_=3221553242, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.589683] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553242, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302041589139, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.589704] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553242) [2025-05-15 09:40:41.589719] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553242, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.590344] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.600455] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.606946] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.607885] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12343855912, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.607913] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.607940] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.607958] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.607971] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.607989] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12343855912, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.608003] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.608018] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.608032] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.608039] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.608054] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.608065] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.608074] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041607858) [2025-05-15 09:40:41.610574] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.619734] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.619754] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.619769] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.619786] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.619799] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.619813] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.619825] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.619836] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.619849] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.620392] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.620376] INFO [SHARE] blacklist_loop_ (ob_server_blacklist.cpp:311) [4090][Blacklist][T0][Y0-0000000000000000-0-0] [lt=89] blacklist_loop exec finished(cost_time=20, is_enabled=true, send_cnt=0) [2025-05-15 09:40:41.620416] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2651303, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.620432] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2651303, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.620445] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.620455] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2651303, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.620467] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.620480] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.620504] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.620517] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.620528] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.620542] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.620556] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041620554, replica_locations:[]}) [2025-05-15 09:40:41.620573] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.620593] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:317, local_retry_times:317, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:41.620668] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=72][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.620678] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:41.620692] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:41.620687] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.620701] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:41.620707] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:41.620722] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:41.620734] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.620779] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11805511256, cache_obj->added_lc()=false, cache_obj->get_object_id()=15840, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.621835] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.621863] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=27][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.621895] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.621910] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.621924] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.621937] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:41.621946] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:41.621958] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:41.621972] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:41.621982] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.621995] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.622006] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.622019] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.622032] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041622031, replica_locations:[]}) [2025-05-15 09:40:41.622048] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.622060] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:41.622071] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:41.622086] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:41.622095] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:41.622105] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:41.622121] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:41.622134] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:41.622143] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:41.622152] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:41.622159] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:41.622168] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:41.622180] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:41.622190] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:41.622199] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:41.622208] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:41.622217] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:41.622230] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:41.622238] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:41.622254] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:41.622264] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.622276] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:41.622286] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:41.622310] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2649372, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=318, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.624903] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.624971] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:41.625007] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:41.626144] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.629505] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.629534] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041629499) [2025-05-15 09:40:41.629567] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.629586] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041629563) [2025-05-15 09:40:41.629597] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302041429505, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.629620] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12343728874, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.629634] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.629652] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.629660] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=127][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302041429477, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.629664] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.629683] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12343901367, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.629699] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.629717] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.632990] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.633377] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553244, support_ssl=false) [2025-05-15 09:40:41.633520] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221553244, sess.client_addr_="172.22.0.1:40198") [2025-05-15 09:40:41.633540] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553244, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.633657] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553245, support_ssl=false) [2025-05-15 09:40:41.633781] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=36] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.633783] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553245, sess.client_addr_="172.22.0.1:40216") [2025-05-15 09:40:41.633801] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de2adb0, session_id:3221553245, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.633825] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.633834] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553246, support_ssl=false) [2025-05-15 09:40:41.633863] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] sm conn init succ(conn.sessid_=3221553246, sess.client_addr_="172.22.0.1:40200") [2025-05-15 09:40:41.633875] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553246, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.633863] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.633895] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=30][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.633912] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.633925] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.633954] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.633975] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.633975] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553244, proxy_sessid=0) [2025-05-15 09:40:41.634012] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=14][errcode=0] server close connection(sessid=3221553244, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.634019] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.634024] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553244, proxy_sessid=0) [2025-05-15 09:40:41.634032] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.634042] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AA-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553244, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.634053] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=29][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.634080] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.634087] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.634099] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.634108] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.634108] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.634102] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553244, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041633431, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.634129] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553244, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.634129] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.634136] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.634143] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553244, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041633431, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.634159] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553244) [2025-05-15 09:40:41.634156] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553245, proxy_sessid=0) [2025-05-15 09:40:41.634154] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.634174] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.634189] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.634190] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=14][errcode=0] server close connection(sessid=3221553245, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.634177] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553244, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.634202] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553245, proxy_sessid=0) [2025-05-15 09:40:41.634205] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.634213] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:41.634220] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553246, proxy_sessid=0) [2025-05-15 09:40:41.634225] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DA-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553245, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.634248] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=14][errcode=0] server close connection(sessid=3221553246, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.634259] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553246, proxy_sessid=0) [2025-05-15 09:40:41.634272] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:41.634279] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909A-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553246, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.634283] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2adb0, session_id:3221553245, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041633431, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.634319] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553246, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302041633431, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.634338] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553245, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.634357] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553246, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.634366] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a1de2adb0, session_id:3221553245, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041633431, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.634383] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553245) [2025-05-15 09:40:41.634404] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553245, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.634432] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553246, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302041633431, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.634445] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553246) [2025-05-15 09:40:41.634454] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553246, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.634800] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221553247, support_ssl=false) [2025-05-15 09:40:41.634858] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] sm conn init succ(conn.sessid_=3221553247, sess.client_addr_="172.22.0.1:40230") [2025-05-15 09:40:41.634873] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553247, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.635031] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.635058] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.635076] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.635090] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.635098] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.635109] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=11][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.635122] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.635135] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553247, proxy_sessid=0) [2025-05-15 09:40:41.635157] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=12][errcode=0] server close connection(sessid=3221553247, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.635169] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553247, proxy_sessid=0) [2025-05-15 09:40:41.635176] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.635182] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AB-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553247, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.635222] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] new mysql sessid created(conn.sessid_=3221553248, support_ssl=false) [2025-05-15 09:40:41.635261] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553248, sess.client_addr_="172.22.0.1:40236") [2025-05-15 09:40:41.635274] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553248, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.635311] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553247, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302041634486, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.635331] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553247, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.635349] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553247, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302041634486, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.635371] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221553247) [2025-05-15 09:40:41.635384] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553247, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.635470] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.635503] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.635531] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.635555] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.635564] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.635571] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.635591] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.635610] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553248, proxy_sessid=0) [2025-05-15 09:40:41.635636] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=14][errcode=0] server close connection(sessid=3221553248, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.635650] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553248, proxy_sessid=0) [2025-05-15 09:40:41.635659] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:41.635671] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DB-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553248, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.635726] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553248, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041634486, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.635749] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553248, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.635760] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553248, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041634486, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.635775] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553248) [2025-05-15 09:40:41.635786] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553248, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.635832] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553249, support_ssl=false) [2025-05-15 09:40:41.635876] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553249, sess.client_addr_="172.22.0.1:40246") [2025-05-15 09:40:41.635891] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553249, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.636070] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.636124] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.636143] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.636162] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.636169] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.636181] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.636194] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.636210] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553249, proxy_sessid=0) [2025-05-15 09:40:41.636229] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=8][errcode=0] server close connection(sessid=3221553249, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.636242] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553249, proxy_sessid=0) [2025-05-15 09:40:41.636249] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.636256] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909B-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553249, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.636313] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553249, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302041635543, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.636344] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221553249, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.636357] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553249, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302041635543, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.636378] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553249) [2025-05-15 09:40:41.636404] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553249, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.636452] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553251, support_ssl=false) [2025-05-15 09:40:41.636451] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553250, support_ssl=false) [2025-05-15 09:40:41.636480] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] sm conn init succ(conn.sessid_=3221553250, sess.client_addr_="172.22.0.1:40270") [2025-05-15 09:40:41.636496] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553251, sess.client_addr_="172.22.0.1:40256") [2025-05-15 09:40:41.636501] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] accept one succ(*s={this:0x7f1a1de2a030, session_id:3221553250, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.636511] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553251, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.636653] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] new mysql sessid created(conn.sessid_=3221553252, support_ssl=false) [2025-05-15 09:40:41.636663] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=23] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.636692] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.636698] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] sm conn init succ(conn.sessid_=3221553252, sess.client_addr_="172.22.0.1:40286") [2025-05-15 09:40:41.636726] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.636724] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553252, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.636755] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.636778] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=22][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.636784] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.636796] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=17][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.636826] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.636823] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.636842] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553250, proxy_sessid=0) [2025-05-15 09:40:41.636847] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.636850] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.636876] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=15][errcode=0] server close connection(sessid=3221553250, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.636883] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.636890] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553250, proxy_sessid=0) [2025-05-15 09:40:41.636898] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.636903] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.636906] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AC-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553250, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.636921] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.636863] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.636936] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.636945] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.636947] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=82][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.636953] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.636969] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.636969] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=21][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.636966] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de2a030, session_id:3221553250, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.636980] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553252, proxy_sessid=0) [2025-05-15 09:40:41.636986] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553250, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.636993] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de2a030, session_id:3221553250, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.637001] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=12][errcode=0] server close connection(sessid=3221553252, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.636997] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=14] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.637006] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553250) [2025-05-15 09:40:41.637012] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553252, proxy_sessid=0) [2025-05-15 09:40:41.637018] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553251, proxy_sessid=0) [2025-05-15 09:40:41.637022] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:41.637021] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553250, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.637032] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909C-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553252, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.637059] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=13][errcode=0] server close connection(sessid=3221553251, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.637072] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553251, proxy_sessid=0) [2025-05-15 09:40:41.637083] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:41.637081] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553252, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.637102] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553252, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.637098] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DC-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553251, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.637114] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553252, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.637130] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553252) [2025-05-15 09:40:41.637154] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221553252, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.637169] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] new mysql sessid created(conn.sessid_=3221553253, support_ssl=false) [2025-05-15 09:40:41.637195] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553251, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.637217] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553253, sess.client_addr_="172.22.0.1:40300") [2025-05-15 09:40:41.637227] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] kill and revert session(conn.sessid_=3221553251, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.637230] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553253, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.637272] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553251, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.637301] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=27] mark session id unused(conn.sessid_=3221553251) [2025-05-15 09:40:41.637328] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] connection close(sessid=3221553251, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.637347] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221553254, support_ssl=false) [2025-05-15 09:40:41.637375] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553254, sess.client_addr_="172.22.0.1:40304") [2025-05-15 09:40:41.637388] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553254, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.637442] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.637467] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.637502] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.637526] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=24][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.637543] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=17][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.637561] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=18][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.637578] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.637598] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553253, proxy_sessid=0) [2025-05-15 09:40:41.637624] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.637636] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553253, proxy_sessid=0) [2025-05-15 09:40:41.637641] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:41.637643] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.637661] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:41.637654] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AD-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553253, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.637679] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:41.637698] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:41.637706] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:41.637720] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:41.637734] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:41.637753] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553254, proxy_sessid=0) [2025-05-15 09:40:41.637772] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=8][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.637786] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553254, proxy_sessid=0) [2025-05-15 09:40:41.637794] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:41.637784] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553253, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.637805] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553253, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.637799] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909D-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553254, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:41.637815] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553253, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.637830] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553253) [2025-05-15 09:40:41.637844] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553253, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.637878] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553254, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:41.637896] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553254, proxy_sessid=0, ret=0) [2025-05-15 09:40:41.637918] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553254, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302041636600, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:41.637933] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553254) [2025-05-15 09:40:41.637943] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553254, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:41.643104] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.650526] INFO [COMMON] replace_fragment_node (ob_kvcache_map.cpp:725) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] Cache replace map node details(ret=0, replace_node_count=0, replace_time=3013, replace_start_pos=6200000, replace_num=100000) [2025-05-15 09:40:41.650549] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3953][TimerWK0_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=10) [2025-05-15 09:40:41.653225] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.662827] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4050][LSUserTblUp6][T0][YB427F000001-000635291D6D172B-0-0] [lt=30][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1001, task={tenant_id:1002, ls_id:{id:1}, add_timestamp:1747300526212290}) [2025-05-15 09:40:41.663336] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.673448] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.683567] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.693679] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.703788] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.707010] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.707924] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.707955] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041707916) [2025-05-15 09:40:41.707968] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302041507865, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.708023] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=26] get wrs ts(ls_id={id:1}, delta=12343956241, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.708045] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.708068] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.708095] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12343956241, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.708109] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.713896] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.722435] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.722459] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2549263, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.722474] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2549263, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.722499] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.722523] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.722534] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.722543] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.722559] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.722576] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041722574, replica_locations:[]}) [2025-05-15 09:40:41.722595] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.722619] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.723911] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.723933] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.723948] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.723970] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.723984] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.723995] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.724009] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.724019] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041724019, replica_locations:[]}) [2025-05-15 09:40:41.724016] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.724085] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2547597, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=319, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.729572] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12344000793, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.729600] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.729624] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.729658] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041729618) [2025-05-15 09:40:41.729673] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302041629607, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.729695] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12343829357, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.729709] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.734140] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.738179] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=0] table not exist(tenant_id=1, database_id=201001, table_name=__all_mview, table_name.ptr()="data_size:11, data:5F5F616C6C5F6D76696577", ret=-5019) [2025-05-15 09:40:41.738229] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=46][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_mview, ret=-5019) [2025-05-15 09:40:41.738244] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=13][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_mview, db_name=oceanbase) [2025-05-15 09:40:41.738262] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=17][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_mview) [2025-05-15 09:40:41.738283] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=16][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:41.738298] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=15][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:41.738319] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_mview' doesn't exist [2025-05-15 09:40:41.738331] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:41.738347] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=15][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:41.738362] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=14][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:41.738377] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=14][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:41.738394] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=15][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:41.738409] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=14][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:41.738425] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=15][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:41.738450] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=13][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:41.738465] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=13][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.738483] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=15][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.738507] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=21][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:41.738523] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=14][errcode=-5019] fail to handle text query(stmt=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4, ret=-5019) [2025-05-15 09:40:41.738540] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=15][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:41.738555] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=13][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.738584] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=22][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:41.738609] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=19][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.738622] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=13][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.738638] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=15][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:41.738665] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=13][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:41.738683] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=16][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.738699] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [5612][T1_TimerWK4_TntSharedTimer][T1][YB427F000001-0006352924AC6112-0-0] [lt=14][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5019, ret=-5019) [2025-05-15 09:40:41.738718] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [5612][T1_TimerWK4_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:41.738732] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [5612][T1_TimerWK4_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:41.738746] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [5612][T1_TimerWK4_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:41.738760] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [5612][T1_TimerWK4_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302041737807, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:41.738777] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [5612][T1_TimerWK4_TntSharedTimer][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:41.742164] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3953][TimerWK0_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=11] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:41.745764] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.751647] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3956][TimerWK3_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=6] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:41.755880] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.756712] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=41][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.756729] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:41.756741] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:41.756761] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:41.756776] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=4159488) [2025-05-15 09:40:41.756812] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=6] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:41.765989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.776103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.786217] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.787831] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=21][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:32, tid:4630}]) [2025-05-15 09:40:41.796282] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.806395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.807087] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:114) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] tenant info is invalid(ret=-4076, query_ts=0, standby_scn={val:0, v:0}, this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:41.807109] INFO [STORAGE.TRANS] query_and_update_last_id (ob_standby_timestamp_service.cpp:128) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21] ObStandbyTimestampService state(*this={inited:true, last_id:-1, tenant_id:1002, epoch:-1, self:"127.0.0.1:2882", switch_to_leader_ts:-1}) [2025-05-15 09:40:41.807119] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.807964] WDIAG [PALF] convert_to_ts (scn.cpp:221) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4016] invalid scn should not convert to ts (val_=18446744073709551615) [2025-05-15 09:40:41.807985] INFO [STORAGE.TRANS] print_stat_ (ob_tenant_weak_read_service.cpp:535) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] [WRS] [TENANT_WEAK_READ_SERVICE] [STAT](tenant_id=1002, server_version={version:{val:18446744073709551615, v:3}, total_part_count:0, valid_inner_part_count:0, valid_user_part_count:0}, server_version_delta=1747302041807962, in_cluster_service=false, cluster_version={val:18446744073709551615, v:3}, min_cluster_version={val:18446744073709551615, v:3}, max_cluster_version={val:18446744073709551615, v:3}, get_cluster_version_err=0, cluster_version_delta=-1, cluster_service_master="0.0.0.0:0", cluster_service_tablet_id={id:226}, post_cluster_heartbeat_count=0, succ_cluster_heartbeat_count=0, cluster_heartbeat_interval=1000000, local_cluster_version={val:0, v:0}, local_cluster_delta=1747302041807962, force_self_check=false, weak_read_refresh_interval=100000) [2025-05-15 09:40:41.808142] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=134] get wrs ts(ls_id={id:1}, delta=12344056791, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.808161] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.808185] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.808201] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.808212] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.808228] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12344056791, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.808242] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.808263] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.808275] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.808284] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.808296] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.808305] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.808314] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041808118) [2025-05-15 09:40:41.816517] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.820304] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=20] gc stale ls task succ [2025-05-15 09:40:41.824211] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.824233] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2447486, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.824248] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2447486, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.824272] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.824287] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.824298] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.824313] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.824330] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041824328, replica_locations:[]}) [2025-05-15 09:40:41.824347] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.824372] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.825525] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.825547] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.825562] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.825585] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.825598] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.825610] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.825626] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.825639] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041825638, replica_locations:[]}) [2025-05-15 09:40:41.825700] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2445981, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=320, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.826634] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.829622] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.829663] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041829615) [2025-05-15 09:40:41.829682] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302041629671, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.829698] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12343928705, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.829717] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12344101198, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.829717] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.829736] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.829755] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.829741] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.829764] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.829776] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.829775] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.829785] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.829792] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041829683) [2025-05-15 09:40:41.829792] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.829806] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.829823] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.829836] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.829848] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041829700) [2025-05-15 09:40:41.836752] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.846864] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.850635] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=29] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=9) [2025-05-15 09:40:41.856982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.862911] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=26] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:41.862940] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:41.862955] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:41.862969] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:41.862982] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:41.862993] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:41.863006] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=7][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:41.863016] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:41.863026] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=10][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:41.863034] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=6][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:41.863041] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=6][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:41.863051] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:41.863059] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=6][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:41.863069] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:41.863088] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:41.863099] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.863108] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=7][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:41.863116] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=7][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:41.863125] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:41.863135] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=8][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:41.863145] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=10][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:41.863160] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=12][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:41.863180] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.863190] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=9][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:41.863196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:41.863212] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:41.863225] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=12][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.863236] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4247][T1_Occam][T1][YB427F000001-000635291C6C6242-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:41.863248] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:41.863260] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:41.863272] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:41.863282] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302041862730, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:41.863296] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:41.863304] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, tenant_id=1, columns=0x7f1abce563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:41.863323] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:41.863407] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1abce563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:41.863423] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:41.863435] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:41.863443] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:41.863461] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=11] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:41.867109] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.877224] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.887335] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.888041] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5627, dropped:36, tid:4602}]) [2025-05-15 09:40:41.893096] INFO [DETECT] record_summary_info_and_logout_when_necessary_ (ob_lcl_batch_sender_thread.cpp:193) [4624][T1002_LCLSender][T1002][Y0-0000000000000000-0-0] [lt=31] ObLCLBatchSenderThread periodic report summary info(duty_ratio_percentage=0, total_constructed_detector=0, total_destructed_detector=0, total_alived_detector=0, _lcl_op_interval=30000, lcl_msg_map_.count()=0, *this={this:0x7f1a903de2b0, is_inited:true, is_running:true, total_record_time:5010000, over_night_times:0}) [2025-05-15 09:40:41.897447] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.907126] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:41.907568] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.908022] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.908046] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041908017) [2025-05-15 09:40:41.908061] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302041707994, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.908096] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12344155891, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.908113] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.908133] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.908145] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.908155] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.908170] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12344156946, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:41.908183] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.908198] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:41.908209] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:41.908221] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:41.908234] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.908243] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:41.908251] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041908079) [2025-05-15 09:40:41.915092] INFO [CLOG] is_in_sync_ (palf_handle_impl.cpp:5323) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=0] is_in_sync(ret=-4721, palf_id=1, is_in_sync=false, is_use_cache=false, remote_check_is_log_sync=false, leader_max_scn={val:18446744073709551615, v:3}, local_max_scn={val:1747289698016024000, v:0}) [2025-05-15 09:40:41.917678] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.919970] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=1][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.919993] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.920013] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.920031] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.920044] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.920059] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.920069] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.920080] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302041919946, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.920093] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.920102] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:41.920169] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:41.920183] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:41.920193] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:41.920203] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:41.925835] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.925861] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2345866, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.925876] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2345866, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.925904] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.925923] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.925938] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.925955] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.925974] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041925971, replica_locations:[]}) [2025-05-15 09:40:41.925994] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:41.926019] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:41.927281] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:41.927303] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.927319] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:41.927341] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.927356] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:41.927364] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:41.927379] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:41.927393] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302041927392, replica_locations:[]}) [2025-05-15 09:40:41.927460] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2344222, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=321, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:41.927738] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4579][T1002_TimerWK2_TenantMetaMemMg][T1002][Y0-0000000000000000-0-0] [lt=10] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:728, free_obj_hold_cnt:0, allocator used:2795520, allocator total:3002880}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=3002880, tablets_mem_limit=7774753240, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:9, free_obj_hold_cnt:0, allocator used:18432, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=728) [2025-05-15 09:40:41.927797] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.929685] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:41.929743] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:41.929765] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302041929736) [2025-05-15 09:40:41.929778] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302041729685, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:41.929716] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] get_min_unreplayed_log_info(lsn={lsn:2944224391}, scn={val:1747289697727835002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300530150701, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:1686016, ret_code_:0}}, next_to_submit_lsn_:{lsn:2944224391}, next_to_submit_scn_:{val:1747289697727835002, v:0}, base_lsn_:{lsn:2818400256}, base_scn_:{val:1747274404592930000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69205000, next_round_pread_size:3678208, curr_read_pos:3609237, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3609237, log_storage_:{start_lsn:{lsn:2940615154}, end_lsn:{lsn:2944224391}, read_buf:{buf_len_:3747840, buf_:0x7f1a69205000}, block_size:67104768, log_storage_:0x7f1ab0f4fbf0, log_storage_:{log_tail:{lsn:2944224980}, readable_log_tail:{lsn:2944224980}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:43, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1/1/log", dir_fd:128, min_block_id:0, max_block_id:44, curr_writable_block_id:43}, logical_block_size_:67104768, curr_block_writable_size_:8384812, block_header_serialize_buf_:0x7f1ab0f508b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697727835001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697727835001, v:0}, data_checksum:2923900223, flag:13900}}, init_mode_version:0, accumulate_checksum:2881944827, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697727835001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:35, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:64, read_io_size:126980096, read_disk_cost_ts:1613509, start_lsn:{lsn:2818400256}}}}}}) [2025-05-15 09:40:41.929803] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12344028942, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:41.929810] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=93] get wrs ts(ls_id={id:1}, delta=12344201435, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:41.929819] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.929823] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:41.929838] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:41.929838] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:41.929850] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.929852] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:41.929858] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.929860] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:41.929870] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.929872] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.929879] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:41.929881] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:41.929888] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041929676) [2025-05-15 09:40:41.929889] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302041929790) [2025-05-15 09:40:41.931851] INFO [PALF] next (palf_iterator.h:183) [4665][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=24] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:41.931855] INFO [PALF] next (palf_iterator.h:183) [4667][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=32] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:41.935952] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=29] PNIO [ratelimit] time: 1747302041935949, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007618, add_bytes: 0 [2025-05-15 09:40:41.937040] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2755) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", [SlidingWindow]={palf_id:1, self:"127.0.0.1:2882", lsn_allocator:{max_log_id:8886935, max_lsn:5311387831, max_scn:{val:1747289698016024000, v:0}}, group_buffer:{log_group_buffer: start_lsn:{lsn:5311387238}, reuse_lsn:{lsn:5311387831}, reserved_buffer_size:41943040, available_buffer_size:41943040, readable_begin_lsn:{lsn:5311387831}}, last_submit_lsn:{lsn:5311387238}, last_submit_end_lsn:{lsn:5311387831}, last_submit_log_id:8886935, last_submit_log_pid:4, max_flushed_lsn:{lsn:5311387238}, max_flushed_end_lsn:{lsn:5311387831}, max_flushed_log_pid:4, committed_end_lsn:{lsn:5311387238}, last_slide_log_id:8886934, last_slide_scn:{val:1747289697900328000, v:0}, last_slide_lsn:{lsn:5311386719}, last_slide_end_lsn:{lsn:5311387238}, last_slide_log_pid:4, last_slide_log_accum_checksum:3961629555, last_fetch_end_lsn:{lsn:18446744073709551615}, last_fetch_max_log_id:-1, last_fetch_committed_end_lsn:{lsn:18446744073709551615}, last_truncate_lsn:{lsn:18446744073709551615}, last_fetch_req_time:0, is_truncating:false, is_rebuilding:false, last_rebuild_lsn:{lsn:18446744073709551615}, freeze_mode:"FEEDBACK_FREEZE_MODE", has_pending_handle_submit_task:false, last_fetch_trigger_type:"LOG_LOOP_TH", this:0x7f1a97bc4530}, [StateMgr]={this:0x7f1a97bc89b0, self:"127.0.0.1:2882", palf_id:1, role:"FOLLOWER", replica_type:"NORMAL_REPLICA", state:"ACTIVE", prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, leader:"0.0.0.0:0", leader_epoch:0, is_sync_enabled:true, allow_vote:true, allow_vote_persisted:true, pending_end_lsn:{lsn:18446744073709551615}, scan_disk_log_finished:true, last_check_start_id:-1, is_changing_config_with_arb:false, reconfirm_start_time_us:-1, palf_role_change_cb:0x7f1a97bcf470, allow_vote:true}, [ConfigMgr]={palf_id:1, self:"127.0.0.1:2882", alive_paxos_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, alive_paxos_replica_num:1, log_ms_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:4427296652}, prev_mode_pid:4}, running_args:{server:{server:"0.0.0.0:0", timestamp:-1, flag:0}, curr_member_list:0, curr_replica_num:0, new_replica_num:0, config_version:{proposal_id:9223372036854775807, config_seq:-1}, ref_scn:{val:18446744073709551615, v:3}, lock_owner:-1, lock_type:0, added_list:0, removed_list:0, new_member_list:0, type:"Invalid"}, state:0, checking_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, reconfig_barrier:{prev_log_proposal_id:9223372036854775807, prev_lsn:{lsn:0}, prev_end_lsn:{lsn:0}, prev_mode_pid:9223372036854775807}, persistent_config_version:{proposal_id:4, config_seq:5}, ms_ack_list:0, resend_config_version:{proposal_id:9223372036854775807, config_seq:-1}, resend_log_list:{learner_num:0, learner_array:[]}, last_submit_config_log_time_us:-1, need_change_config_bkgd:false, bkgd_config_version:{proposal_id:9223372036854775807, config_seq:-1}, region:"default_region", register_time_us:-1, parent:"0.0.0.0:0", parent_keepalive_time_us:-1, last_submit_register_req_time_us:-1, children:{learner_num:0, learner_array:[]}, last_submit_keepalive_time_us:-1, this:0x7f1a97bc5c70}, [ModeMgr]={palf_id:1, self:"127.0.0.1:2882", applied_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, accepted_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, last_submit_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}, state:"MODE_INIT", new_proposal_id:9223372036854775807, local_max_lsn:{lsn:18446744073709551615}, local_max_log_pid:9223372036854775807, max_majority_accepted_pid:9223372036854775807, max_majority_lsn:{lsn:18446744073709551615}, max_majority_accepted_mode_meta:{version:-1, proposal_id:9223372036854775807, mode_version:9223372036854775807, access_mode:0, ref_scn:{val:18446744073709551615, v:3}}, follower_list:0, ack_list:0, majority_cnt:9223372036854775807, last_submit_req_ts:-1, resend_mode_meta_list:{learner_num:0, learner_array:[]}}, [LogEngine]={palf_id:1, is_inited:true, min_block_max_scn:{val:18446744073709551615, v:3}, min_block_id:274894685184, min_block_min_scn:{val:18446744073709551615, v:3}, base_lsn_for_block_gc:{lsn:5167067136}, log_meta:{version:1, log_prepare_meta:{version:1, voted_for:"0.0.0.0:0", log_proposal_id:4}, log_config_meta:{version:3, proposal_id:4, prev:{version:1, config:{config_version:{proposal_id:3, config_seq:4}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, curr:{version:1, config:{config_version:{proposal_id:4, config_seq:5}, log_sync_memberlist:1{server:"172.19.0.2:2882", timestamp:1, flag:0}, log_sync_replica_num:1, arbitration_member:{server:"0.0.0.0:0", timestamp:-1, flag:0}, learnerlist:{learner_num:0, learner_array:[]}, degraded_learnerlist:{learner_num:0, learner_array:[]}}, lock_meta:{version:1, lock_type:0, lock_owner:-1, lock_time:-1}}, prev_log_proposal_id:3, prev_lsn:{lsn:4427296652}, prev_mode_pid:4}, log_snapshot_meta:{version:2, base_lsn:{lsn:5167067136}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, log_replica_property_meta:{allow_vote:true, replica_type:"NORMAL_REPLICA"}, log_mode_meta:{version:1, proposal_id:4, mode_version:0, access_mode:1, ref_scn:{val:1, v:0}}}, log_meta_storage:{log_tail:{lsn:118784}, readable_log_tail:{lsn:118784}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:5368381440}, min_scn:{val:0, v:0}, curr_block_id:0, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/meta", dir_fd:131, min_block_id:0, max_block_id:1, curr_writable_block_id:0}, logical_block_size_:67104768, curr_block_writable_size_:66985984, block_header_serialize_buf_:0x7f1a97bcabb8, flashback_version:0}, log_storage:{log_tail:{lsn:5311387831}, readable_log_tail:{lsn:5311387831}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:79, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:133, min_block_id:61, max_block_id:80, curr_writable_block_id:79}, logical_block_size_:67104768, curr_block_writable_size_:56993609, block_header_serialize_buf_:0x7f1a97bcc8b8, flashback_version:0}, palf_epoch:1, last_purge_throttling_ts:-1, this:0x7f1a97bc90f0}, [Reconfirm]={palf_id:1, self:"127.0.0.1:2882", state:"INITED", new_proposal_id:9223372036854775807, prepare_log_ack_list:0, curr_paxos_follower_list:0, majority_cnt:0, majority_max_log_server:"0.0.0.0:0", majority_max_accept_pid:9223372036854775807, majority_max_lsn:{lsn:18446744073709551615}, saved_end_lsn:{lsn:18446744073709551615}, last_submit_prepare_req_time_us:-1, last_fetch_log_time_us:-1, last_record_sw_start_id:-1, last_notify_fetch_time_us:-1, last_purge_throttling_time_us:-1, this:0x7f1a97bc8b30}) [2025-05-15 09:40:41.937240] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=14] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4847631358, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:41.937265] WDIAG [LIB] try_wait (thread.cpp:239) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] pthread_tryjoin_np failed(pret=16, errno=0, ret=-4023) [2025-05-15 09:40:41.937267] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=224][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:41.937274] INFO [LIB] do_thread_recycle (threads.cpp:141) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=9] try_wait return eagain(this=0x7f1ae73fe3b0, thread=0x7f1aaefec3a0) [2025-05-15 09:40:41.937282] INFO [PALF] check_and_switch_state (palf_handle_impl.cpp:2759) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=14] [PALF_DUMP](palf_id=1, self="127.0.0.1:2882", ack_info_list=[]) [2025-05-15 09:40:41.937285] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=10] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4847631358, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:41.937296] WDIAG [PALF] get_ack_info_array (log_sliding_window.cpp:2760) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4018] match_lsn_map_ get failed(ret=-4018, palf_id=1, self="127.0.0.1:2882", tmp_server={server:"172.19.0.2:2882", timestamp:1, flag:0}) [2025-05-15 09:40:41.937308] INFO [PALF] report_log_task_trace (log_sliding_window.cpp:4312) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12] current log_task status(palf_id=1, self="127.0.0.1:2882", log_id=8886935, log_task={header:{begin_lsn:{lsn:5311387238}, end_lsn:{lsn:5311387831}, log_id:8886935, min_scn:{val:1747289698016024000, v:0}, max_scn:{val:1747289698016024000, v:0}, data_len:537, proposal_id:4, prev_lsn:{lsn:5311386719}, prev_proposal_id:4, committed_end_lsn:{lsn:5311387238}, data_checksum:2749435567, accum_checksum:2227909955, is_padding_log:false, is_raw_write:false}, state_map:{val:26}, ref_cnt:0, gen_ts:1747300521030351, freeze_ts:1747300521030351, submit_ts:-1, flushed_ts:-1, gen_to_freeze cost time:0, gen_to_submit cost time:-1747300521030352, submit_to_flush cost time:0}, ack_info_list=[]) [2025-05-15 09:40:41.937341] WDIAG [STORAGE] get_server_region (ob_locality_manager.cpp:396) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4018] fail to get server region(ret=-4018, server="127.0.0.1:2882") [2025-05-15 09:40:41.937352] WDIAG [CLOG] get_server_region (ob_locality_adapter.cpp:61) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4018] get_server_region failed(server="127.0.0.1:2882") [2025-05-15 09:40:41.937362] WDIAG [PALF] update_self_region_ (palf_handle_impl.cpp:398) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4018] get_server_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:41.937373] WDIAG [PALF] check_and_switch_state (palf_handle_impl.cpp:2766) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4018] update_region failed(ret=-4018, this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}) [2025-05-15 09:40:41.937930] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.941829] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=16] PNIO [ratelimit] time: 1747302041941827, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007611, add_bytes: 0 [2025-05-15 09:40:41.948035] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.951908] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=22] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:41.958146] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.959612] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=33] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:41.959637] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] temporary file module use no blocks [2025-05-15 09:40:41.959648] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:41.959665] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:41.959680] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=14] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11804850035, flush_io_finished_ret_:0, flush_io_finished_round_:1516, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:41.962580] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=22] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:41.968258] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.978366] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.980908] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:41.988258] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=22][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:41, tid:3961}]) [2025-05-15 09:40:41.988473] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.996010] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEA-0-0] [lt=6][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.996029] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEA-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.996050] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEA-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.996067] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEA-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.996079] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CEA-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.996091] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:41.996104] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.996116] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.996127] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302041995985, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:41.996137] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.996146] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:41.996221] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:41.997461] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEB-0-0] [lt=11][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.997478] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEB-0-0] [lt=16][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:41.997495] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEB-0-0] [lt=7][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:41.997509] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEB-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:41.997519] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CEB-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:41.997535] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:41.997545] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:41.997554] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:41.997565] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302041997451, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:41.997577] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:41.997583] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:41.997622] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:41.998588] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:41.998978] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:42.007181] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.007434] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4578][T1002_TimerWK1_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.007716] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=51][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.007741] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.007750] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.007805] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.007813] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:42.007826] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:42.007911] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:42.007931] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:42.007975] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:42.008098] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12344256274, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.008122] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.008145] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.008162] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.008174] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.008191] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12344256274, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.008203] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.008217] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.008228] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.008237] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.008250] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.008259] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.008268] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042008075) [2025-05-15 09:40:42.008696] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.012236] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4559][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=19] ====== tenant freeze timer task ====== [2025-05-15 09:40:42.018811] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.020123] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.020144] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.020166] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.020183] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.020196] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.020210] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.020224] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.020234] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.020244] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302042020095, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.020257] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.020267] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.020340] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:42.020353] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:42.020363] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:42.020372] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:42.020589] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:102) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=57] tx gc loop thread is running(MTL_ID()=1002) [2025-05-15 09:40:42.020644] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:109) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=54] try gc retain ctx [2025-05-15 09:40:42.020653] INFO [STORAGE.TRANS] run1 (ob_tx_loop_worker.cpp:116) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=9] try to retry start_working [2025-05-15 09:40:42.020694] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=14] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:1747289693216794002, v:0}, keep_alive_scn_:{val:1747289693216794002, v:0}, update_ts_:1747302042020153}) [2025-05-15 09:40:42.021294] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=27] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:42.021392] INFO [STORAGE] update_min_start_scn_info (ob_tx_table.cpp:1036) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=84] finish update min start scn(max_decided_scn={val:1747289697751194001, v:0}, ctx_min_start_scn_info_={min_start_scn_in_ctx_:{val:0, v:0}, keep_alive_scn_:{val:0, v:0}, update_ts_:0}) [2025-05-15 09:40:42.021982] INFO [STORAGE.TRANS] do_tx_gc_ (ob_tx_loop_worker.cpp:274) [4638][T1002_TxLoopWor][T1002][Y0-0000000000000000-0-0] [lt=16] [Tx Loop Worker] check tx scheduler success(MTL_ID()=1002, *ls_ptr={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:42.027591] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.027615] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2244108, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.027632] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2244108, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.027680] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=48][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.027690] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2244108, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.027706] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.027724] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.027739] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.027755] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.027767] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.027783] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.027800] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042027798, replica_locations:[]}) [2025-05-15 09:40:42.027819] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.027844] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.027856] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:42.027880] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.027928] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11805918405, cache_obj->added_lc()=false, cache_obj->get_object_id()=15844, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.028933] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.029172] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.029193] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.029209] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.029220] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.029232] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.029245] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.029260] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.029277] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.029291] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.029302] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.029316] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.029330] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042029330, replica_locations:[]}) [2025-05-15 09:40:42.029401] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2242281, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=322, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.029660] INFO [STORAGE.TRANS] dump_mapper_info (ob_lock_wait_mgr.h:67) [4606][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=18] report RowHolderMapper summary info(count=0, bkt_cnt=248) [2025-05-15 09:40:42.029744] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.029774] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042029735) [2025-05-15 09:40:42.029792] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302041829700, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.029798] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.029810] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:42.029828] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:42.029839] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12344301811, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.029856] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.029875] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.029889] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.029899] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.029912] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.029921] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.029853] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get_min_unreplayed_log_info(lsn={lsn:5311387238}, scn={val:1747289697900328001, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747300529803179, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:5311387238}, next_to_submit_scn_:{val:1747289697900328001, v:0}, base_lsn_:{lsn:5167067136}, base_scn_:{val:1747274408663046000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a72405000, next_round_pread_size:3678208, curr_read_pos:2755105, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2755105, log_storage_:{start_lsn:{lsn:5308632133}, end_lsn:{lsn:5311387238}, read_buf:{buf_len_:3747840, buf_:0x7f1a72405000}, block_size:67104768, log_storage_:0x7f1a97bcbbf0, log_storage_:{log_tail:{lsn:5311387831}, readable_log_tail:{lsn:5311387831}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:79, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1001/1/log", dir_fd:133, min_block_id:61, max_block_id:80, curr_writable_block_id:79}, logical_block_size_:67104768, curr_block_writable_size_:56993609, block_header_serialize_buf_:0x7f1a97bcc8b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697900328000, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:431, scn_:{val:1747289697900328000, v:0}, data_checksum:4152786848, flag:25505}}, init_mode_version:0, accumulate_checksum:3961629555, curr_entry_is_padding:0, padding_entry_size:463, padding_entry_scn:{val:1747289697900328000, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:41, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:76, read_io_size:145543168, read_disk_cost_ts:2159331, start_lsn:{lsn:5167067136}}}}}}) [2025-05-15 09:40:42.029931] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042029827) [2025-05-15 09:40:42.029936] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=83] get wrs ts(ls_id={id:1}, delta=12344129318, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.029948] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.029963] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.029976] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.029985] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.029998] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.030006] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.030014] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042029820) [2025-05-15 09:40:42.037075] INFO [PALF] log_loop_ (log_loop_thread.cpp:154) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=10] LogLoopThread round_cost_time(us)(round_cost_time=376) [2025-05-15 09:40:42.039043] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.049166] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.050746] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=34] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=8) [2025-05-15 09:40:42.059288] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.066196] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=18] ====== check clog disk timer task ====== [2025-05-15 09:40:42.066217] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4580][T1002_TimerWK3_CKClogDisk][T1002][Y0-0000000000000000-0-0] [lt=17] get_disk_usage(ret=0, capacity(MB):=13824, used(MB):=1731) [2025-05-15 09:40:42.069396] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.070742] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=8] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:42.070772] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.070797] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:42.070808] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:42.070818] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:42.070899] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:42.074289] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.074330] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:42.074356] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:42.074343] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC96BD-0-0] [lt=28][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302042074313}) [2025-05-15 09:40:42.074370] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:42.074385] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:42.074742] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.074755] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:42.074768] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:42.079504] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.088472] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=19][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4076, dropped:9, tid:4542}]) [2025-05-15 09:40:42.089611] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.090277] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553263, support_ssl=false) [2025-05-15 09:40:42.090355] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553263, sess.client_addr_="172.22.0.1:40318") [2025-05-15 09:40:42.090375] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553263, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.090641] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.090686] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.090718] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.090739] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.090752] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.090763] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.090788] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.090806] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553263, proxy_sessid=0) [2025-05-15 09:40:42.090850] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=14][errcode=0] server close connection(sessid=3221553263, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.090863] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553263, proxy_sessid=0) [2025-05-15 09:40:42.090870] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.090881] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DD-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553263, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.090937] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553263, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302042089805, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.090961] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553263, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.090988] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=26] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553263, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302042089805, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.091005] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553263) [2025-05-15 09:40:42.091024] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553263, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.099716] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.107239] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.108061] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4269][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=27] ====== tenant freeze timer task ====== [2025-05-15 09:40:42.108139] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.108173] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042108132) [2025-05-15 09:40:42.108187] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302041908074, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.108225] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12344356601, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.108244] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.108266] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.108281] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.108291] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.108308] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12344356601, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.108320] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.108334] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.108347] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.108354] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.108367] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.108379] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.108389] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042108202) [2025-05-15 09:40:42.109828] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.116076] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:42.116095] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:42.116104] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=8] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:42.116121] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:42.119941] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.120162] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.120181] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.120204] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.120220] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.120234] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.120245] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.120256] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.120267] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.120278] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302042120137, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.120290] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.120301] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.120377] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:42.120388] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:42.120397] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:42.120407] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:42.120505] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=28] gc stale ls task succ [2025-05-15 09:40:42.129195] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=30] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:42.129222] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:42.129236] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=12] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:42.129254] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:42.129269] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11805019617, flush_io_finished_ret_:0, flush_io_finished_round_:1516, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:42.129534] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.129557] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2142162, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.129573] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2142162, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.129584] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.129596] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2142162, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.129611] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.129628] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.129642] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.129656] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.129667] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.129679] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.129695] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042129693, replica_locations:[]}) [2025-05-15 09:40:42.129722] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=26] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.129746] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.129758] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:42.129806] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.129812] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12344401096, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.129835] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.129850] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11806020329, cache_obj->added_lc()=false, cache_obj->get_object_id()=15845, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.129854] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.129858] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.129876] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.129874] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042129849) [2025-05-15 09:40:42.129888] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.129888] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302042029819, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.129905] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.129910] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12344228603, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.129914] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.129921] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042129795) [2025-05-15 09:40:42.129924] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.129940] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.129953] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.129962] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.129975] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.129987] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.129993] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042129898) [2025-05-15 09:40:42.130095] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=28][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.130968] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.130989] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.131004] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.131019] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.131032] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.131045] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.131060] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.131074] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.131087] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.131099] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.131114] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.131124] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042131123, replica_locations:[]}) [2025-05-15 09:40:42.131187] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2140494, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=323, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.132415] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4289][T1_TimerWK0_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=22] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:42.132449] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=20] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:42.134952] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] new mysql sessid created(conn.sessid_=3221553265, support_ssl=false) [2025-05-15 09:40:42.135023] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=32] sm conn init succ(conn.sessid_=3221553265, sess.client_addr_="172.22.0.1:40330") [2025-05-15 09:40:42.135042] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553265, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.135116] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221553266, support_ssl=false) [2025-05-15 09:40:42.135245] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553266, sess.client_addr_="172.22.0.1:40332") [2025-05-15 09:40:42.135247] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.135270] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] accept one succ(*s={this:0x7f1a1de04030, session_id:3221553266, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.135295] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.135317] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221553267, support_ssl=false) [2025-05-15 09:40:42.135354] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553267, sess.client_addr_="172.22.0.1:40348") [2025-05-15 09:40:42.135352] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.135371] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553267, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.135394] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=40][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.135417] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=22][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.135432] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.135456] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=28] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.135460] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.135478] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553265, proxy_sessid=0) [2025-05-15 09:40:42.135508] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.135529] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=26][errcode=0] server close connection(sessid=3221553265, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.135544] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553265, proxy_sessid=0) [2025-05-15 09:40:42.135553] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.135538] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.135572] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=32][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.135568] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AE-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553265, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.135587] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.135592] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.135601] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=9][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.135616] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.135620] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.135634] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553266, proxy_sessid=0) [2025-05-15 09:40:42.135643] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.135650] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553265, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302042134215, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.135667] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=15][errcode=0] server close connection(sessid=3221553266, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.135669] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.135674] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] kill and revert session(conn.sessid_=3221553265, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.135677] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553266, proxy_sessid=0) [2025-05-15 09:40:42.135684] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.135686] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:42.135684] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553265, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302042134215, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.135696] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.135700] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553265) [2025-05-15 09:40:42.135700] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909E-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553266, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.135719] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.135738] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553267, proxy_sessid=0) [2025-05-15 09:40:42.135724] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553265, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.135767] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=10][errcode=0] server close connection(sessid=3221553267, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.135780] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553267, proxy_sessid=0) [2025-05-15 09:40:42.135788] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.135802] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DE-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553267, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.135802] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221553266, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042135271, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.135852] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=44] kill and revert session(conn.sessid_=3221553266, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.135849] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553267, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302042135271, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.135870] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553267, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.135869] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221553266, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042135271, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.135877] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553267, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302042135271, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.135893] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] mark session id unused(conn.sessid_=3221553266) [2025-05-15 09:40:42.135894] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553267) [2025-05-15 09:40:42.135908] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553267, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.135921] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553266, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.136170] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553268, support_ssl=false) [2025-05-15 09:40:42.136209] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553268, sess.client_addr_="172.22.0.1:40362") [2025-05-15 09:40:42.136225] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553268, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.136391] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.136420] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.136447] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.136471] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.136486] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.136512] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=27][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.136529] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.136545] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553268, proxy_sessid=0) [2025-05-15 09:40:42.136573] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=13][errcode=0] server close connection(sessid=3221553268, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.136585] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553268, proxy_sessid=0) [2025-05-15 09:40:42.136592] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.136605] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90AF-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553268, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.136658] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553268, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042136336, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.136682] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553268, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.136691] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553268, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042136336, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.136709] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553268) [2025-05-15 09:40:42.136720] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553268, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.136800] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553269, support_ssl=false) [2025-05-15 09:40:42.136897] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] sm conn init succ(conn.sessid_=3221553269, sess.client_addr_="172.22.0.1:40372") [2025-05-15 09:40:42.136913] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221553269, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.137030] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.137067] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.137084] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.137102] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.137109] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.137123] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=14][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.137135] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.137150] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553269, proxy_sessid=0) [2025-05-15 09:40:42.137167] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=8][errcode=0] server close connection(sessid=3221553269, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.137181] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553269, proxy_sessid=0) [2025-05-15 09:40:42.137188] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.137201] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC909F-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553269, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.137251] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221553269, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302042136336, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.137275] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553269, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.137287] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221553269, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302042136336, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.137304] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553269) [2025-05-15 09:40:42.137319] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553269, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.137505] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221553270, support_ssl=false) [2025-05-15 09:40:42.137565] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] sm conn init succ(conn.sessid_=3221553270, sess.client_addr_="172.22.0.1:40374") [2025-05-15 09:40:42.137585] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de06030, session_id:3221553270, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.137800] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.137838] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.137855] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=24][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.137873] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.137880] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.137893] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.137907] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.137923] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553270, proxy_sessid=0) [2025-05-15 09:40:42.137940] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=8][errcode=0] server close connection(sessid=3221553270, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.137952] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553270, proxy_sessid=0) [2025-05-15 09:40:42.137960] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:42.137967] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90DF-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553270, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.138015] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de06030, session_id:3221553270, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302042137395, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.138040] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553270, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.138047] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de06030, session_id:3221553270, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302042137395, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.138065] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553270) [2025-05-15 09:40:42.138078] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553270, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.138198] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553271, support_ssl=false) [2025-05-15 09:40:42.138241] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553271, sess.client_addr_="172.22.0.1:40388") [2025-05-15 09:40:42.138258] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553271, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.138285] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553272, support_ssl=false) [2025-05-15 09:40:42.138321] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553272, sess.client_addr_="172.22.0.1:40392") [2025-05-15 09:40:42.138337] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1a9d9fedb0, session_id:3221553272, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.138473] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.138513] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=25] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.138533] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.138550] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=7] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.138549] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.138568] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.138571] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] new mysql sessid created(conn.sessid_=3221553273, support_ssl=false) [2025-05-15 09:40:42.138578] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=27][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.138586] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.138592] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.138596] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.138603] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.138607] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.138612] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221553273, sess.client_addr_="172.22.0.1:40402") [2025-05-15 09:40:42.138624] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.138624] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a9d9fe030, session_id:3221553273, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.138631] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.138636] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553272, proxy_sessid=0) [2025-05-15 09:40:42.138651] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553271, proxy_sessid=0) [2025-05-15 09:40:42.138661] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=14][errcode=0] server close connection(sessid=3221553272, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.138670] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553272, proxy_sessid=0) [2025-05-15 09:40:42.138675] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=9][errcode=0] server close connection(sessid=3221553271, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.138683] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=13][errcode=-4016] session is null [2025-05-15 09:40:42.138688] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553271, proxy_sessid=0) [2025-05-15 09:40:42.138691] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A0-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553272, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.138699] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=10][errcode=-4016] session is null [2025-05-15 09:40:42.138712] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B0-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553271, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.138737] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fedb0, session_id:3221553272, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.138771] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553271, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.138793] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553272, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.138802] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221553271, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.138816] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a9d9fedb0, session_id:3221553272, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.138834] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553272) [2025-05-15 09:40:42.138848] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553272, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.138870] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553271, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.138888] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=18] mark session id unused(conn.sessid_=3221553271) [2025-05-15 09:40:42.138898] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553271, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.138939] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.138925] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=34] new mysql sessid created(conn.sessid_=3221553274, support_ssl=false) [2025-05-15 09:40:42.138965] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.138977] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.138984] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] sm conn init succ(conn.sessid_=3221553274, sess.client_addr_="172.22.0.1:40408") [2025-05-15 09:40:42.138989] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] new mysql sessid created(conn.sessid_=3221553275, support_ssl=false) [2025-05-15 09:40:42.138995] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.139003] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.139010] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.139003] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553274, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.139022] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.139026] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553275, sess.client_addr_="172.22.0.1:40414") [2025-05-15 09:40:42.139037] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553273, proxy_sessid=0) [2025-05-15 09:40:42.139039] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221553275, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:148}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.139055] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=8][errcode=0] server close connection(sessid=3221553273, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.139066] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553273, proxy_sessid=0) [2025-05-15 09:40:42.139073] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.139085] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E0-0-0] [lt=12] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553273, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.139129] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fe030, session_id:3221553273, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.139151] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553273, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.139159] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a9d9fe030, session_id:3221553273, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.139178] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553273) [2025-05-15 09:40:42.139189] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553273, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.139207] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.139225] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.139245] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=10] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.139261] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.139265] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=20] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.139277] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=16][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.139286] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.139280] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=23][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.139294] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.139300] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.139309] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.139315] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=15][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.139322] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553275, proxy_sessid=0) [2025-05-15 09:40:42.139324] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.139340] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=8][errcode=0] server close connection(sessid=3221553275, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.139339] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.139352] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553275, proxy_sessid=0) [2025-05-15 09:40:42.139354] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553274, proxy_sessid=0) [2025-05-15 09:40:42.139363] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:42.139378] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=12][errcode=0] server close connection(sessid=3221553274, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.139373] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A1-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553275, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.139392] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553274, proxy_sessid=0) [2025-05-15 09:40:42.139399] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.139411] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B1-0-0] [lt=11] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553274, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.139420] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221553275, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.139444] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553275, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.139458] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221553275, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:148}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.139455] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553274, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.139478] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553275) [2025-05-15 09:40:42.139482] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553274, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.139499] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553275, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.139505] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553274, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302042138452, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.139524] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553274) [2025-05-15 09:40:42.139546] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553274, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.140206] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.150320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.152178] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=29] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44876530483, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:42.160435] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.170558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.174251] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.174457] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.174998] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=19] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.175099] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=16] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.180670] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.190777] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.192189] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=18] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:42.200884] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.203284] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=16] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:42.203314] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=26][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:42.203327] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=12][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:42.203340] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=12][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:42.203351] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=8][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:42.203358] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:42.203375] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=11][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:42.203386] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=10][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:42.203396] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=9][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:42.203405] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=8][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:42.203415] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=9][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:42.203426] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:42.203437] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=9][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:42.203446] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=9][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.203517] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4460][T1001_TimerWK0_LSMetaCh][T1][YB427F000001-000635291CFC6458-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.207314] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.208207] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:42.208245] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=32] get_min_unreplayed_log_info(lsn={lsn:934287681}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302042132217, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:934287681}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:872361984}, base_scn_:{val:1747274406063927000, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a69605000, next_round_pread_size:3678208, curr_read_pos:3075493, curr_read_buf_start_pos:0, curr_read_buf_end_pos:3075493, log_storage_:{start_lsn:{lsn:931212188}, end_lsn:{lsn:934287681}, read_buf:{buf_len_:3747840, buf_:0x7f1a69605000}, block_size:67104768, log_storage_:0x7f1a73fd1bf0, log_storage_:{log_tail:{lsn:934288498}, readable_log_tail:{lsn:934288498}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:13, palf_id:1}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1/log", dir_fd:143, min_block_id:0, max_block_id:14, curr_writable_block_id:13}, logical_block_size_:67104768, curr_block_writable_size_:5178254, block_header_serialize_buf_:0x7f1a73fd28b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:2923900223, flag:16486}}, init_mode_version:0, accumulate_checksum:910368143, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}}}}) [2025-05-15 09:40:42.208339] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=90] get wrs ts(ls_id={id:1}, delta=12344457086, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.208364] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.208386] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.208401] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.208412] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.208428] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12344457086, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.208440] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.208459] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.208468] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.208475] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.208496] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.208510] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.208519] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042208192) [2025-05-15 09:40:42.210989] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.220095] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.220115] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.220132] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.220150] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.220165] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.220177] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.220189] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.220198] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.220210] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302042220078, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.220223] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.220232] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.220285] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:42.220295] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:42.220305] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:42.220316] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:42.221110] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.229859] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.229884] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042229852) [2025-05-15 09:40:42.229901] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302042029804, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.229916] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.229948] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042229913) [2025-05-15 09:40:42.229963] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302042129898, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.229986] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12344329089, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.229999] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.230015] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.230027] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.230035] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.230046] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.230053] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.230051] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=138] get wrs ts(ls_id={id:1}, delta=12344501582, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.230059] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042229973) [2025-05-15 09:40:42.230064] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.230081] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.230094] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.230103] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.230117] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.230127] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.230135] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042230037) [2025-05-15 09:40:42.231223] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.231318] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.231341] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2040383, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.231358] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2040383, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.231373] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.231386] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2040383, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.231403] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.231418] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.231434] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.231448] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.231460] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.231476] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.231505] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=23] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042231503, replica_locations:[]}) [2025-05-15 09:40:42.231524] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.231550] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.231562] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:42.231590] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.231637] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11806122115, cache_obj->added_lc()=false, cache_obj->get_object_id()=15846, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.232885] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.232905] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.232922] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.232935] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.232943] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.232957] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.232972] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.232985] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.232999] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.233011] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.233026] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.233040] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042233039, replica_locations:[]}) [2025-05-15 09:40:42.233105] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=2038577, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=324, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.241329] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.250837] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=21] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=7) [2025-05-15 09:40:42.251442] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.256861] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.256879] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:42.256891] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:42.256911] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:42.256929] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=6239232) [2025-05-15 09:40:42.256961] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:42.261556] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.271167] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=23] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:42.271665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.281772] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.285529] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6454-0-0] [lt=19][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.285559] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6454-0-0] [lt=29][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.285611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6454-0-0] [lt=37][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.285629] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6454-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.285642] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6454-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1001'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.285655] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:42.285664] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.285675] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.285686] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302042285503, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:42.285701] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.285711] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1001') [2025-05-15 09:40:42.285791] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1001, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:42.285838] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=48][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:42.287073] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6455-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.287090] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6455-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.287104] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6455-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.287116] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6455-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.287123] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4462][T1001_TimerWK2_FreInfoReload][T1001][YB427F000001-000635291DBC6455-0-0] [lt=6][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.287134] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:42.287144] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.287154] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.287163] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302042287063, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:42.287173] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.287181] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute sql failed(ret=-5627, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:42.287220] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to select for update snapshot_gc_scn(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1001) [2025-05-15 09:40:42.287231] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to load updated info(ret=-5627) [2025-05-15 09:40:42.287241] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4462][T1001_TimerWK2_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] fail to try update info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:42.292255] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.302940] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.307383] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.308254] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.308287] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042308247) [2025-05-15 09:40:42.308302] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302042108201, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.308337] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12344556479, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.308357] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.308381] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.308398] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.308419] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.308431] INFO [CLOG] get_max_applied_scn (ob_log_apply_service.cpp:739) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get_max_applied_scn(scn={val:18446744073709551615, v:3}, this={ls_id_:{id:1001}, role_:2, proposal_id_:-1, palf_committed_end_lsn_:{lsn:0}, palf_committed_end_scn_:{val:18446744073709551615, v:3}, last_check_scn_:{val:18446744073709551615, v:3}, max_applied_cb_scn_:{val:18446744073709551615, v:3}}) [2025-05-15 09:40:42.308451] INFO [CLOG] get_min_unreplayed_log_info (ob_replay_status.cpp:1038) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get_min_unreplayed_log_info(lsn={lsn:747348431}, scn={val:1747289697751194002, v:0}, this={ls_id_:{id:1001}, is_enabled_:true, is_submit_blocked_:false, role_:2, err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false, err_ts_:0, err_ret_:0}, ref_cnt_:2, post_barrier_lsn_:{lsn:18446744073709551615}, pending_task_count_:0, submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1747302042232468, err_info_:{has_fatal_error_:false, fail_ts_:0, fail_cost_:0, ret_code_:0}}, next_to_submit_lsn_:{lsn:747348431}, next_to_submit_scn_:{val:1747289697751194002, v:0}, base_lsn_:{lsn:738152448}, base_scn_:{val:1747289697751194001, v:0}, iterator_:{iterator_impl:{buf_:0x7f1a5a205000, next_round_pread_size:3678208, curr_read_pos:1839725, curr_read_buf_start_pos:0, curr_read_buf_end_pos:1839725, log_storage_:{start_lsn:{lsn:745508706}, end_lsn:{lsn:747348431}, read_buf:{buf_len_:3747840, buf_:0x7f1a5a205000}, block_size:67104768, log_storage_:0x7f1a73f61bf0, log_storage_:{log_tail:{lsn:747348553}, readable_log_tail:{lsn:747348553}, log_block_header:{magic:18754, version:1, min_lsn:{lsn:18446744073709551615}, min_scn:{val:18446744073709551615, v:3}, curr_block_id:11, palf_id:1001}, block_mgr:{log_dir:"/root/ob/observer/store/clog/tenant_1002/1001/log", dir_fd:140, min_block_id:0, max_block_id:12, curr_writable_block_id:11}, logical_block_size_:67104768, curr_block_writable_size_:57908663, block_header_serialize_buf_:0x7f1a73f628b8, flashback_version:0}, storage_type:"DISK_STORAGE"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1747289697751194001, v:0}, curr_entry:{LogEntryHeader:{magic:19528, version:2, log_size:34, scn_:{val:1747289697751194001, v:0}, data_checksum:154833137, flag:60041}}, init_mode_version:0, accumulate_checksum:2356512705, curr_entry_is_padding:0, padding_entry_size:66, padding_entry_scn:{val:1747289697751194001, v:0}}, io_ctx:{user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}}}}) [2025-05-15 09:40:42.308542] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=92] get wrs ts(ls_id={id:1001}, delta=12344556479, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.308553] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.308568] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.308579] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.308588] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.308601] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.308618] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.308624] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042308314) [2025-05-15 09:40:42.308691] WDIAG load_file_to_string (utility.h:673) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=18][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:42.308710] WDIAG get_ethernet_speed (utility.cpp:575) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:42.308723] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=12][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:42.308732] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3954][TimerWK1_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:42.313051] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.313950] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=14][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:42.313974] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=24][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.313990] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=10][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.314079] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=89][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.314093] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.314106] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.314121] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.314139] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.314157] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.314171] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.314182] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.314196] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.314224] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=11] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.314237] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=10] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:42.315827] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=13] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:42.315855] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=24][errcode=-5019] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.315932] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6392-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.320961] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.321026] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=65][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.321070] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.321101] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=24][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.321138] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=33][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.321163] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.321191] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=27][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.321213] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.321227] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302042320897, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.321248] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.321270] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.321524] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=23][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:42.321561] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=36][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:42.321579] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:42.321597] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:42.323199] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.329946] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12344601228, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.329979] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.330000] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12344428735, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.330013] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=30][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.330025] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.330042] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=27][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.330043] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.330055] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.330054] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.330063] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.330074] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.330072] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.330085] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.330086] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.330093] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042329983) [2025-05-15 09:40:42.330099] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042329914) [2025-05-15 09:40:42.333270] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.333294] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.333316] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.333332] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.333343] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.333360] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.333379] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.333398] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.333416] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.333428] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.333445] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.333467] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042333464, replica_locations:[]}) [2025-05-15 09:40:42.333475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=63][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.333498] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.333530] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.333541] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:42.333591] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.333657] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11806224129, cache_obj->added_lc()=false, cache_obj->get_object_id()=15847, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.335153] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.335174] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.335190] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.335204] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.335212] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.335227] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.335242] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.335256] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.335270] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.335282] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.335296] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.335310] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042335309, replica_locations:[]}) [2025-05-15 09:40:42.335379] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=1936303, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=325, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.343612] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=30][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.352462] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=33] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44874433331, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:42.353729] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.356090] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7] start do ls ha handler(ls_id_array_=[{id:1}, {id:1001}]) [2025-05-15 09:40:42.356122] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.356136] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:42.356147] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:42.356157] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:42.356239] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:42.356251] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.356263] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:42.356272] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:42.356279] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:42.356329] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=50][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:42.357717] INFO [COMMON] clean_garbage_node (ob_kvcache_map.cpp:668) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=42] Cache wash clean map node details(ret=0, clean_node_count=0, clean_time=5212, clean_start_pos=600000, clean_num=200000) [2025-05-15 09:40:42.363843] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.369346] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:42.369449] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=28] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:42.369526] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=20] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:42.369587] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:42.369616] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:16101, total:16112) [2025-05-15 09:40:42.369652] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=10] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:42.369712] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=54] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:16101, total:16112) [2025-05-15 09:40:42.369737] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:42.369758] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:42.369770] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=9] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:42.369791] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=7] successfully alloc block(macro_id={[ver=1,mode=0,seq=628777][2nd=1904]}) [2025-05-15 09:40:42.373951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.384057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.389094] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=33][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-5019, dropped:66, tid:4289}]) [2025-05-15 09:40:42.394159] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.395447] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=15] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.395998] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=14] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.396091] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.396179] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=14] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.396300] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.396370] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.396475] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:42.404280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.407432] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.408337] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12344657074, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.408365] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.408388] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.408405] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.408415] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.408429] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1001}, delta=12344657074, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.408441] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.408455] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.408467] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.408476] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.408496] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.408506] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.408515] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042408313) [2025-05-15 09:40:42.414395] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.415188] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4461][T1001_TimerWK1_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.416960] INFO [PALF] try_recycle_blocks (palf_env_impl.cpp:799) [4292][T1_TimerWK3_PalfGC][T1][Y0-0000000000000000-0-0] [lt=0] LOG_DISK_OPTION(disk_options_wrapper_={disk_opts_for_stopping_writing:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):5120, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):183, sequence:1}) [2025-05-15 09:40:42.420419] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.420438] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.420464] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.420482] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.420503] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.420515] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.420528] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.420540] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.420552] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302042420384, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.420565] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.420576] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.420654] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:42.420666] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:42.420678] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:42.420688] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:42.424511] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.429975] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:42.429993] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.430028] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042429968) [2025-05-15 09:40:42.430034] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.430045] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302042230037, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.430053] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042430030) [2025-05-15 09:40:42.430067] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302042229972, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.430068] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12344701575, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.430084] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.430095] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12344529082, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.430107] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.430104] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.430123] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.430124] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.430133] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.430135] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.430143] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.430147] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.430155] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.430155] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.430164] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.430163] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042430055) [2025-05-15 09:40:42.430172] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042430078) [2025-05-15 09:40:42.434623] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.435482] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.435514] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.435531] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.435545] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.435558] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.435574] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.435591] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.435606] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.435622] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.435630] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.435645] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.435663] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042435661, replica_locations:[]}) [2025-05-15 09:40:42.435681] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.435705] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.435717] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:42.435746] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.435795] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11806326273, cache_obj->added_lc()=false, cache_obj->get_object_id()=15848, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.436928] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.436948] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.436966] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.436980] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.436988] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.437002] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.437017] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.437031] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.437046] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.437054] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.437070] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.437085] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042437084, replica_locations:[]}) [2025-05-15 09:40:42.437146] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=1834536, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=326, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.444732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.450928] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3956][TimerWK3_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=6) [2025-05-15 09:40:42.454846] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.464952] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.474366] WDIAG [ARCHIVE] do_thread_task_ (ob_archive_sender.cpp:232) [4501][T1001_ArcSender][T1001][YB427F000001-000635291CBC6112-0-0] [lt=20][errcode=-4018] try free send task failed(ret=-4018) [2025-05-15 09:40:42.476103] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.486216] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.487597] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4289][T1_TimerWK0_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.489316] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=27][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:409, tid:4482}]) [2025-05-15 09:40:42.496332] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.506438] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.507506] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=33][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.507529] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:42.507635] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=106][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.507655] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:42.507681] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:42.507696] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:42.507710] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:42.507725] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:42.507746] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:42.508024] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.508044] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.508055] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.508065] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.508074] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:42.508087] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:42.508102] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:42.508113] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=8] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:42.508147] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:42.508365] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.508389] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042508360) [2025-05-15 09:40:42.508403] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302042308313, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.508431] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12344756443, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.508448] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.508468] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.508483] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.508502] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.508517] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1001}, delta=12344756443, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.508529] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.508542] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.508553] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.508562] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.508574] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.508585] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.508593] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042508416) [2025-05-15 09:40:42.509389] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=20][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.509409] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.509420] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.509431] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=11][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.509442] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=10][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:42.509459] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=9][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:42.509473] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.509496] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_TntSharedTimer][T1002][YB427F000001-000635291F9C666C-0-0] [lt=13][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4"}, aret=-5157, ret=-5157) [2025-05-15 09:40:42.509510] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:42.509521] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.509531] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:42.509539] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78cb440, start=1747302042509369, sql=SELECT CAST(MVIEW_ID AS UNSIGNED) AS MVIEW_ID, LAST_REFRESH_SCN, CAST(REFRESH_MODE AS UNSIGNED) AS REFRESH_MODE FROM `oceanbase`.`__all_mview` WHERE TENANT_ID = 0 and REFRESH_MODE = 4) [2025-05-15 09:40:42.509552] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:42.509683] INFO [STORAGE] runTimerTask (ob_tenant_restore_info_mgr.cpp:281) [4578][T1002_TimerWK1_TntSharedTimer][T1002][Y0-0000000000000000-0-0] [lt=14] refresh restore info [2025-05-15 09:40:42.514048] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6679-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.514430] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6679-0-0] [lt=385][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.514453] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6679-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.514471] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_FreInfoReload][T1001][YB427F000001-000635291F6C6679-0-0] [lt=17][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.514499] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C6679-0-0] [lt=28][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_merge_info WHERE tenant_id = '1002'"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.514511] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:42.514524] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.514537] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.514551] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302042514022, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:42.514569] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.514584] WDIAG [SHARE] load_global_merge_info (ob_global_merge_table_operator.cpp:45) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to execute sql(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, meta_tenant_id=1001, sql=SELECT * FROM __all_merge_info WHERE tenant_id = '1002') [2025-05-15 09:40:42.514653] WDIAG [STORAGE] refresh_merge_info (ob_tenant_freeze_info_mgr.cpp:593) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5627] failed to load global merge info(ret=-5627, ret="OB_SCHEMA_EAGAIN", global_merge_info={tenant_id:1002, cluster:{name:"cluster", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, frozen_scn:{name:"frozen_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, global_broadcast_scn:{name:"global_broadcast_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, last_merged_scn:{name:"last_merged_scn", is_scn:true, scn:{val:1, v:0}, value:-1, need_update:false}, is_merge_error:{name:"is_merge_error", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_status:{name:"merge_status", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, error_type:{name:"error_type", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, suspend_merging:{name:"suspend_merging", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, merge_start_time:{name:"merge_start_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}, last_merged_time:{name:"last_merged_time", is_scn:false, scn:{val:18446744073709551615, v:3}, value:0, need_update:false}}) [2025-05-15 09:40:42.514700] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:725) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=47][errcode=-5627] fail to refresh merge info(tmp_ret=-5627, tmp_ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:42.515937] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.515959] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=22][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.515967] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=7][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.515978] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=10][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.515989] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=10][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:42.516001] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=8][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:42.516012] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.516021] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4578][T1002_TimerWK1_FreInfoReload][T1002][YB427F000001-000635291F6C667A-0-0] [lt=9][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, aret=-5157, ret=-5157) [2025-05-15 09:40:42.516031] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:42.516041] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:42.516049] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:42.516058] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78cada0, start=1747302042515925, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:42.516068] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:42.516077] WDIAG [SHARE] inner_get_snapshot_gc_scn_ (ob_global_stat_proxy.cpp:627) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5157] execute sql failed(ret=-5157, sql=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' ) [2025-05-15 09:40:42.516117] WDIAG [SHARE] fetch_new_freeze_info (ob_freeze_info_manager.cpp:181) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to select for update snapshot_gc_scn(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.516129] WDIAG [STORAGE] try_update_info (ob_tenant_freeze_info_mgr.cpp:655) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] failed to load updated info(ret=-5157) [2025-05-15 09:40:42.516141] WDIAG [STORAGE] runTimerTask (ob_tenant_freeze_info_mgr.cpp:728) [4578][T1002_TimerWK1_FreInfoReload][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to try update info(tmp_ret=-5157, tmp_ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:42.516558] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.520195] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.520214] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.520226] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.520239] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.520249] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.520261] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.520280] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=19][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.520292] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.520306] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302042520183, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.520323] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.520339] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.520399] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:42.520412] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:42.520421] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:42.520431] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:42.525433] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.525460] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.525501] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=39][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.526680] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.526718] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:42.526755] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=37][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.526774] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.526799] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:42.526811] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:42.526823] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:42.530058] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9] get wrs ts(ls_id={id:1}, delta=12344802026, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.530084] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.530104] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.530102] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.530120] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.530130] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.530126] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042530090) [2025-05-15 09:40:42.530140] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302042430078, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.530142] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.530151] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.530158] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042530037) [2025-05-15 09:40:42.530161] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12344629533, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.530176] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.530201] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.530220] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.530228] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.530238] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.530247] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.530256] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042530150) [2025-05-15 09:40:42.536793] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.537300] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.537320] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.537337] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.537351] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.537362] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.537378] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.537394] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.537397] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:42.537408] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.537423] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.537423] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=26][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.537434] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.537432] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.537450] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.537466] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042537464, replica_locations:[]}) [2025-05-15 09:40:42.537484] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.537513] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:326, local_retry_times:326, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:42.537535] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.537548] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:42.537563] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:42.537572] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:42.537581] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:42.537611] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:42.537625] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.537676] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11806428151, cache_obj->added_lc()=false, cache_obj->get_object_id()=15849, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.538934] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.538967] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=31][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.539007] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.539023] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.539038] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.539051] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.539062] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.539076] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.539090] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.539108] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.539125] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.539140] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.539155] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.539169] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042539167, replica_locations:[]}) [2025-05-15 09:40:42.539185] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.539197] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.539209] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.539226] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:42.539237] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:42.539247] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:42.539264] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:42.539279] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:42.539286] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:42.539296] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:42.539306] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:42.539315] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:42.539327] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:42.539338] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:42.539348] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:42.539357] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:42.539367] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:42.539378] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:42.539390] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:42.539405] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:42.539416] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:42.539426] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:42.539436] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:42.539448] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:42.539458] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=327, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:42.539486] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] will sleep(sleep_us=100000, remain_us=1732196, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=327, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.542580] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=19] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:42.545666] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=17] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:42.546909] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.557015] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.558011] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3955][TimerWK2_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=18] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44872336179, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:42.567123] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.575198] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.575222] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.575248] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1, GTS_LS={id:1}) [2025-05-15 09:40:42.575264] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:42.575282] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:42.575301] WDIAG [STORAGE.TRANS] operator() (ob_ts_mgr.h:176) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] refresh gts failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:42.575313] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=12] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:42.575321] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:42.575329] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.575351] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:42.575369] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:42.575385] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:42.575396] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1001) [2025-05-15 09:40:42.575405] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1001, GTS_LS={id:1}) [2025-05-15 09:40:42.575415] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:42.575430] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:42.575786] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:42.575801] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.575816] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.575833] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:150) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4721] check if ls exist failed(tmp_ret=-5157, ls_id={id:1}) [2025-05-15 09:40:42.575841] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:42.575851] WDIAG [STORAGE.TRANS] get_leader_ (ob_location_adapter.cpp:154) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] failed to check tenant has been dropped(tmp_ret=-4006, tenant_id=1002) [2025-05-15 09:40:42.575857] WDIAG [STORAGE.TRANS] get_gts_leader_ (ob_gts_source.cpp:529) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4721] gts nonblock get leader failed(ret=-4721, tenant_id=1002, GTS_LS={id:1}) [2025-05-15 09:40:42.575865] WDIAG [STORAGE.TRANS] refresh_gts_ (ob_gts_source.cpp:588) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4721] get gts leader failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:42.575880] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=9] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:42.577237] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.582841] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C666D-0-0] [lt=13][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.582866] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C666D-0-0] [lt=24][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.582888] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C666D-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.582904] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C666D-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.582942] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=36][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.582954] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:42.582967] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.582979] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.582991] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=11][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302042582818, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:42.583008] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.583018] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:42.583088] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=12][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:42.583103] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=13][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:42.583117] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=13][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:42.583128] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=10][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:42.583143] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=12][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:42.583156] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=10][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:42.583168] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C666D-0-0] [lt=11][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:42.587345] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.592071] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553288, support_ssl=false) [2025-05-15 09:40:42.592199] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553288, sess.client_addr_="172.22.0.1:40424") [2025-05-15 09:40:42.592219] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de04030, session_id:3221553288, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.592443] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.592480] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.592524] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.592598] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=36][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.592620] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.592632] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.592657] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.592678] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553288, proxy_sessid=0) [2025-05-15 09:40:42.592720] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=13][errcode=0] server close connection(sessid=3221553288, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.592731] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553288, proxy_sessid=0) [2025-05-15 09:40:42.592740] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.592754] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E1-0-0] [lt=10] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553288, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.592818] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221553288, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042592258, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.592845] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553288, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.592857] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221553288, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042592258, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.592875] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553288) [2025-05-15 09:40:42.592896] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553288, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.593852] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4463][T1001_TimerWK3_TenantMetaMemMg][T1001][Y0-0000000000000000-0-0] [lt=17] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:990, free_obj_hold_cnt:20, allocator used:3878400, allocator total:4177920}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:14, free_obj_hold_cnt:0, allocator used:918274, allocator total:918274}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5096194, tablets_mem_limit=863861460, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:25, free_obj_hold_cnt:0, allocator used:51200, allocator total:65280}, pending_cnt=0, wait_gc_count=0, tablet count=1004) [2025-05-15 09:40:42.597456] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.607562] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=31][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.607570] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.608427] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.608468] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=30][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042608419) [2025-05-15 09:40:42.608484] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302042508414, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.608529] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18] get wrs ts(ls_id={id:1}, delta=12344856935, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.608549] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.608572] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.608590] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.608675] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=85][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.608690] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1001}, delta=12344856935, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.608705] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.608725] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.608742] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.608752] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.608767] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.608778] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.608788] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042608503) [2025-05-15 09:40:42.617674] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.620353] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.620380] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=26][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.620402] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.620420] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.620454] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=33][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.620469] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.620481] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.620509] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=25][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.620525] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302042620328, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:42.620541] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.625495] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.625517] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.625530] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.626776] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:42.626803] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=25][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.626823] WDIAG [CLOG] nonblock_get_leader (ob_location_adapter.cpp:127) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4721] location adapter nonblock_get_leader failed(ret=-4721, id=1, leader="0.0.0.0:0") [2025-05-15 09:40:42.626838] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.626859] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:42.626871] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:42.626886] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:42.627786] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.630089] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.630116] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042630085) [2025-05-15 09:40:42.630132] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302042430055, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.630162] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12344901423, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.630163] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12344728930, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.630180] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.630181] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.630201] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.630202] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.630214] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.630215] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.630222] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.630221] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.630233] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.630234] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.630242] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.630243] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.630249] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042630150) [2025-05-15 09:40:42.630250] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042630147) [2025-05-15 09:40:42.636809] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553290, support_ssl=false) [2025-05-15 09:40:42.636897] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] sm conn init succ(conn.sessid_=3221553290, sess.client_addr_="172.22.0.1:40438") [2025-05-15 09:40:42.636914] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553290, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.636954] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27] new mysql sessid created(conn.sessid_=3221553292, support_ssl=false) [2025-05-15 09:40:42.636934] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553291, support_ssl=false) [2025-05-15 09:40:42.636998] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553292, sess.client_addr_="172.22.0.1:40458") [2025-05-15 09:40:42.637012] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221553292, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.637027] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=51] sm conn init succ(conn.sessid_=3221553291, sess.client_addr_="172.22.0.1:40446") [2025-05-15 09:40:42.637055] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=24] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553291, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.637124] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.637161] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.637257] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.637265] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=40] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.637194] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.637287] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.637301] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=103][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.637312] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=18] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.637321] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.637335] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.637343] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.637359] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.637325] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.637369] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.637375] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=14][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.637386] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.637388] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=63][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.637390] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553290, proxy_sessid=0) [2025-05-15 09:40:42.637401] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.637412] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.637408] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.637431] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=12][errcode=0] server close connection(sessid=3221553290, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.637432] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.637434] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=23][errcode=-4018] get session fail(ret=-4018, sessid=3221553291, proxy_sessid=0) [2025-05-15 09:40:42.637446] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553290, proxy_sessid=0) [2025-05-15 09:40:42.637447] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553292, proxy_sessid=0) [2025-05-15 09:40:42.637454] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.637458] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:42.637470] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=12][errcode=0] server close connection(sessid=3221553291, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.637470] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=10][errcode=0] server close connection(sessid=3221553292, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.637480] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553292, proxy_sessid=0) [2025-05-15 09:40:42.637465] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A2-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553290, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.637475] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.637480] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553291, proxy_sessid=0) [2025-05-15 09:40:42.637504] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=24][errcode=-4016] session is null [2025-05-15 09:40:42.637508] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=32][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.637510] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=29][errcode=-4016] session is null [2025-05-15 09:40:42.637513] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B2-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553292, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.637517] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E2-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553291, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.637548] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553290, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042636655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.637561] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221553292, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302042636655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.637588] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=36] kill and revert session(conn.sessid_=3221553290, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.637575] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553291, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302042636655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.637595] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=31] kill and revert session(conn.sessid_=3221553292, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.637608] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221553292, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302042636655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.637606] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553290, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042636655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.637609] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=30] kill and revert session(conn.sessid_=3221553291, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.637627] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553292) [2025-05-15 09:40:42.637627] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] mark session id unused(conn.sessid_=3221553290) [2025-05-15 09:40:42.637626] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553291, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302042636655, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.637641] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553292, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.637655] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] mark session id unused(conn.sessid_=3221553291) [2025-05-15 09:40:42.637653] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553290, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.637679] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=14] connection close(sessid=3221553291, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.637770] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553293, support_ssl=false) [2025-05-15 09:40:42.637807] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553293, sess.client_addr_="172.22.0.1:40464") [2025-05-15 09:40:42.637826] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553293, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.637997] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.638027] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.638049] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=38][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.638072] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=21][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.638081] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.638088] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.638110] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.638126] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553293, proxy_sessid=0) [2025-05-15 09:40:42.638147] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=9][errcode=0] server close connection(sessid=3221553293, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.638150] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] new mysql sessid created(conn.sessid_=3221553294, support_ssl=false) [2025-05-15 09:40:42.638160] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553293, proxy_sessid=0) [2025-05-15 09:40:42.638167] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.638175] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A3-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553293, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.638202] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553294, sess.client_addr_="172.22.0.1:40466") [2025-05-15 09:40:42.638221] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de0edb0, session_id:3221553294, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.638231] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553293, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042637713, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.638251] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] kill and revert session(conn.sessid_=3221553293, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.638259] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553293, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042637713, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.638276] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553293) [2025-05-15 09:40:42.638287] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553293, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.638399] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.638431] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.638446] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.638459] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.638470] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.638480] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.638509] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=22] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.638522] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553294, proxy_sessid=0) [2025-05-15 09:40:42.638544] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=10][errcode=0] server close connection(sessid=3221553294, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.638554] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553294, proxy_sessid=0) [2025-05-15 09:40:42.638566] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:42.638572] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E3-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553294, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.638621] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0edb0, session_id:3221553294, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302042637713, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.638649] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24] kill and revert session(conn.sessid_=3221553294, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.638664] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de0edb0, session_id:3221553294, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302042637713, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.638682] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553294) [2025-05-15 09:40:42.638694] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553294, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.638949] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553295, support_ssl=false) [2025-05-15 09:40:42.638954] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=1] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=72791040) [2025-05-15 09:40:42.638990] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553295, sess.client_addr_="172.22.0.1:40470") [2025-05-15 09:40:42.638993] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=30] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:42.639003] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553295, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.639195] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.639219] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.639230] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=22][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.639242] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=13][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.639253] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.639259] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.639274] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=10] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.639287] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553295, proxy_sessid=0) [2025-05-15 09:40:42.639308] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=12][errcode=0] server close connection(sessid=3221553295, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.639318] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553295, proxy_sessid=0) [2025-05-15 09:40:42.639325] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.639334] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B3-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553295, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.639377] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553295, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042638770, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.639397] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553295, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.639407] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553295, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042638770, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.639425] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553295) [2025-05-15 09:40:42.639436] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553295, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.639634] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553296, support_ssl=false) [2025-05-15 09:40:42.639626] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.639648] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.639666] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.639669] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553296, sess.client_addr_="172.22.0.1:40474") [2025-05-15 09:40:42.639680] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.639682] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de0e030, session_id:3221553296, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.639693] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.639708] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.639724] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.639738] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.639752] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.639763] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.639776] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.639793] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042639792, replica_locations:[]}) [2025-05-15 09:40:42.639810] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.639829] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:327, local_retry_times:327, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:42.639849] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.639861] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:42.639876] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:42.639874] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.639886] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:42.639893] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:42.639898] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.639910] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:42.639911] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553297, support_ssl=false) [2025-05-15 09:40:42.639923] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.639921] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=27][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.639947] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=25][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.639949] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553297, sess.client_addr_="172.22.0.1:40490") [2025-05-15 09:40:42.639961] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.639968] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.639962] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553297, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.639974] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11806530448, cache_obj->added_lc()=false, cache_obj->get_object_id()=15850, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.639987] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640002] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553296, proxy_sessid=0) [2025-05-15 09:40:42.640031] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=13][errcode=0] server close connection(sessid=3221553296, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640050] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553296, proxy_sessid=0) [2025-05-15 09:40:42.640061] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.640063] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:42.640081] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A4-0-0] [lt=17] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553296, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.640149] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=8] new mysql sessid created(conn.sessid_=3221553298, support_ssl=false) [2025-05-15 09:40:42.640199] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.640221] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=41] sm conn init succ(conn.sessid_=3221553298, sess.client_addr_="172.22.0.1:40502") [2025-05-15 09:40:42.640247] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=19] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.640244] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a1de1edb0, session_id:3221553298, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.640289] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.640296] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de0e030, session_id:3221553296, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.640322] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=32][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.640329] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=30] kill and revert session(conn.sessid_=3221553296, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.640342] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=20][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.640348] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=18] can close safely, do destroy(*s={this:0x7f1a1de0e030, session_id:3221553296, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.640361] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=18][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.640369] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553296) [2025-05-15 09:40:42.640388] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640399] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] connection close(sessid=3221553296, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.640413] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=23][errcode=-4018] get session fail(ret=-4018, sessid=3221553297, proxy_sessid=0) [2025-05-15 09:40:42.640449] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=19][errcode=0] server close connection(sessid=3221553297, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640452] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221553299, support_ssl=false) [2025-05-15 09:40:42.640460] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.640469] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=18][errcode=-4018] get session fail(ret=-4018, sessid=3221553297, proxy_sessid=0) [2025-05-15 09:40:42.640500] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=17][errcode=-4016] session is null [2025-05-15 09:40:42.640517] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553299, sess.client_addr_="172.22.0.1:40514") [2025-05-15 09:40:42.640519] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.640514] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E4-0-0] [lt=27] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553297, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.640535] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.640536] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a6f604db0, session_id:3221553299, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.640548] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.640557] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.640567] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.640568] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21] new mysql sessid created(conn.sessid_=3221553300, support_ssl=false) [2025-05-15 09:40:42.640566] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553297, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.640585] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=13] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640587] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553297, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.640600] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553298, proxy_sessid=0) [2025-05-15 09:40:42.640601] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=13] sm conn init succ(conn.sessid_=3221553300, sess.client_addr_="172.22.0.1:40520") [2025-05-15 09:40:42.640602] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553297, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.640619] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553297) [2025-05-15 09:40:42.640615] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a6f616030, session_id:3221553300, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.640623] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=13][errcode=0] server close connection(sessid=3221553298, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640632] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553298, proxy_sessid=0) [2025-05-15 09:40:42.640640] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:42.640641] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553297, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.640648] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B4-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553298, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.640710] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1edb0, session_id:3221553298, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.640736] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.640746] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=32] kill and revert session(conn.sessid_=3221553298, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.640759] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=13] can close safely, do destroy(*s={this:0x7f1a1de1edb0, session_id:3221553298, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.640767] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.640776] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553298) [2025-05-15 09:40:42.640780] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=26][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.640799] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.640795] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] connection close(sessid=3221553298, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.640812] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.640824] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.640838] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:42.640841] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640852] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553299, proxy_sessid=0) [2025-05-15 09:40:42.640864] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=13] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:42.640870] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=7][errcode=0] server close connection(sessid=3221553299, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640880] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553299, proxy_sessid=0) [2025-05-15 09:40:42.640877] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=25][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:42.640891] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=11][errcode=-4016] session is null [2025-05-15 09:40:42.640895] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:42.640897] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A5-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553299, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.640909] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=13][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:42.640916] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:42.640929] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640944] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553300, proxy_sessid=0) [2025-05-15 09:40:42.640938] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604db0, session_id:3221553299, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.640959] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553299, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.640968] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=13][errcode=0] server close connection(sessid=3221553300, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:42.640968] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a6f604db0, session_id:3221553299, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.640979] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553300, proxy_sessid=0) [2025-05-15 09:40:42.640983] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553299) [2025-05-15 09:40:42.640986] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:42.640993] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E5-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553300, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:42.640997] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553299, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.641030] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f616030, session_id:3221553300, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:42.641050] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] kill and revert session(conn.sessid_=3221553300, proxy_sessid=0, ret=0) [2025-05-15 09:40:42.641070] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] can close safely, do destroy(*s={this:0x7f1a6f616030, session_id:3221553300, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302042639825, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:42.641088] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553300) [2025-05-15 09:40:42.641099] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553300, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:42.641170] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.641201] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=29][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.641238] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.641254] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.641269] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.641282] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:42.641292] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:42.641305] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:42.641319] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:42.641332] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.641344] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.641355] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.641368] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.641381] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042641380, replica_locations:[]}) [2025-05-15 09:40:42.641396] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.641408] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.641420] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.641434] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:42.641444] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:42.641452] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:42.641468] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:42.641482] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:42.641507] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=24][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:42.641517] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:42.641526] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:42.641535] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:42.641547] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:42.641559] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:42.641569] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:42.641578] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:42.641588] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:42.641597] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:42.641607] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:42.641621] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:42.641632] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:42.641641] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:42.641651] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:42.641663] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:42.641671] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=328, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:42.641695] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14] will sleep(sleep_us=100000, remain_us=1629987, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=328, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.650169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.651021] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3955][TimerWK2_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=28] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=5) [2025-05-15 09:40:42.660280] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.663506] WDIAG [SERVER] batch_process_tasks (ob_ls_table_updater.cpp:322) [4043][LSMetaTblUp0][T0][YB427F000001-000635291B3EB0EA-0-0] [lt=27][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1, task={tenant_id:1, ls_id:{id:1}, add_timestamp:1747302042314220}) [2025-05-15 09:40:42.670386] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.680500] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.690619] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.700732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.707614] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.708512] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12344957300, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.708541] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=25] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.708565] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.708584] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.708596] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.708612] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12344957300, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.708625] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.708640] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.708652] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.708660] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.708673] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.708683] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.708690] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042708476) [2025-05-15 09:40:42.710842] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.720504] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.720951] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.725544] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.725566] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.725583] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.726836] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:42.726861] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.726943] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=80][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.726962] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:42.726974] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:42.726989] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1001) [2025-05-15 09:40:42.730159] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12345001777, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.730185] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.730209] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.730223] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.730214] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.730248] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042730207) [2025-05-15 09:40:42.730263] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302042530149, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.730287] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12344829284, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.730301] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.730318] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.731060] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.737516] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}) [2025-05-15 09:40:42.737535] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1001, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:42.737546] WDIAG [PALF] check_and_try_fetch_log_ (log_state_mgr.cpp:1095) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] sw try_fetch_log failed(ret=-4721, palf_id=1) [2025-05-15 09:40:42.741169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.741818] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.741839] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.741854] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.741868] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.741887] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.741900] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.741911] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.741924] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.741939] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042741937, replica_locations:[]}) [2025-05-15 09:40:42.741955] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.741973] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:328, local_retry_times:328, err_:-4721, err_:"OB_LS_LOCATION_NOT_EXIST", retry_type:1, client_ret:-4721}, need_retry=true) [2025-05-15 09:40:42.741993] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.742011] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:42.742021] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] result set close failed(ret=-4721) [2025-05-15 09:40:42.742027] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=-4721] failed to close result(close_ret=-4721, ret=-4721) [2025-05-15 09:40:42.742053] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to process record(executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, record_ret=-4721, ret=-4721) [2025-05-15 09:40:42.742245] INFO [STORAGE.BLKMGR] runTimerTask (ob_block_manager.cpp:1623) [3955][TimerWK2_BlkMgr][T0][Y0-0000000000000000-0-0] [lt=33] skip inspect bad block(last_check_time=1747300517524757, last_macro_idx=-1) [2025-05-15 09:40:42.743171] WDIAG [SHARE.LOCATION] nonblock_get (ob_location_service.cpp:138) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4721] fail to nonblock get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.743200] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:969) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=28][errcode=-4721] fail to get tablet locations(ret=-4721, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.743237] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.743251] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.743265] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.743285] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.743298] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.743309] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.743330] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.743342] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042743341, replica_locations:[]}) [2025-05-15 09:40:42.743357] WDIAG [SHARE.LOCATION] renew_location_ (ob_ls_location_service.cpp:1017) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] get empty location from meta table(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.743369] WDIAG [SHARE.LOCATION] get (ob_ls_location_service.cpp:280) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4721] renew location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.743380] WDIAG [SHARE.LOCATION] get (ob_location_service.cpp:55) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4721] fail to get log stream location(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}, expire_renew_time=9223372036854775807, is_cache_hit=false, location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}) [2025-05-15 09:40:42.743395] WDIAG [SQL.DAS] block_renew_tablet_location (ob_das_location_router.cpp:1354) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4721] failed to get location(ls_id={id:1}, ret=-4721) [2025-05-15 09:40:42.743405] WDIAG [SQL.DAS] nonblock_get (ob_das_location_router.cpp:981) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] block renew tablet location failed(tmp_ret=-4721, tmp_ret="OB_LS_LOCATION_NOT_EXIST", tablet_id={id:1}) [2025-05-15 09:40:42.743415] WDIAG [SQL.DAS] nonblock_get_candi_tablet_locations (ob_das_location_router.cpp:1014) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Get partition error, the location cache will be renewed later(ret=-4721, tablet_id={id:1}, candi_tablet_loc={opt_tablet_loc:{partition_id:-1, tablet_id:{id:0}, ls_id:{id:-1}, replica_locations:[]}, selected_replica_idx:-1, priority_replica_idxs:[]}) [2025-05-15 09:40:42.743431] WDIAG [SQL.OPT] calculate_candi_tablet_locations (ob_table_location.cpp:1575) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4721] Failed to set partition locations(ret=-4721, partition_ids=[1], tablet_ids=[{id:1}]) [2025-05-15 09:40:42.743444] WDIAG [SQL.OPT] calculate_phy_table_location_info (ob_table_partition_info.cpp:95) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4721] Failed to calculate table location(ret=-4721) [2025-05-15 09:40:42.743453] WDIAG [SQL.JO] compute_table_location (ob_join_order.cpp:261) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calculate table location(ret=-4721) [2025-05-15 09:40:42.743463] WDIAG [SQL.JO] compute_base_table_property (ob_join_order.cpp:10621) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to calc table location(ret=-4721) [2025-05-15 09:40:42.743471] WDIAG [SQL.JO] generate_base_table_paths (ob_join_order.cpp:10562) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to compute base path property(ret=-4721) [2025-05-15 09:40:42.743480] WDIAG [SQL.JO] generate_normal_base_table_paths (ob_join_order.cpp:10548) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate access paths(ret=-4721) [2025-05-15 09:40:42.743501] WDIAG [SQL.OPT] generate_plan_tree (ob_log_plan.cpp:6796) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4721] failed to generate the access path for the single-table query(ret=-4721, get_optimizer_context().get_query_ctx()->get_sql_stmt()=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn') [2025-05-15 09:40:42.743511] WDIAG [SQL.OPT] generate_raw_plan_for_plain_select (ob_select_log_plan.cpp:5228) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] failed to generate plan tree for plain select(ret=-4721) [2025-05-15 09:40:42.743519] WDIAG [SQL.OPT] generate_raw_plan (ob_log_plan.cpp:11175) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4721] fail to generate normal raw plan(ret=-4721) [2025-05-15 09:40:42.743528] WDIAG [SQL.OPT] generate_plan (ob_log_plan.cpp:11132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] fail to generate raw plan(ret=-4721) [2025-05-15 09:40:42.743537] WDIAG [SQL.OPT] optimize (ob_optimizer.cpp:53) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to perform optimization(ret=-4721) [2025-05-15 09:40:42.743546] WDIAG [SQL] optimize_stmt (ob_sql.cpp:3802) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] Failed to optimize logical plan(ret=-4721) [2025-05-15 09:40:42.743556] WDIAG [SQL] generate_plan (ob_sql.cpp:3433) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to optimizer stmt(ret=-4721) [2025-05-15 09:40:42.743570] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3220) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] failed to generate plan(ret=-4721) [2025-05-15 09:40:42.743580] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] Failed to generate plan(ret=-4721, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:42.743591] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle physical plan(ret=-4721) [2025-05-15 09:40:42.743601] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] fail to handle text query(stmt=SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' , ret=-4721) [2025-05-15 09:40:42.743613] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4721] executor execute failed(ret=-4721) [2025-05-15 09:40:42.743622] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4721] execute failed(ret=-4721, tenant_id=1, executor={ObIExecutor:, sql:"SELECT column_value FROM __all_core_table WHERE TABLE_NAME = '__all_global_stat' AND COLUMN_NAME = 'snapshot_gc_scn' "}, retry_cnt=329, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:42.743645] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] will sleep(sleep_us=100000, remain_us=1528037, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=329, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.747079] INFO [SQL.DTL] runTimerTask (ob_dtl_interm_result_manager.cpp:678) [4461][T1001_TimerWK1_TntSharedTimer][T1001][Y0-0000000000000000-0-0] [lt=0] clear dtl interm result cost(us)(clear_cost=21, ret=0, gc_.expire_keys_.count()=0, dump count=0, clean count=0) [2025-05-15 09:40:42.752732] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.756532] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:766) [4463][T1001_TimerWK3_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=43] success to update min reserved snapshot(reserved_snapshot=0, duration=1800, snapshot_gc_ts=0) [2025-05-15 09:40:42.756554] INFO [STORAGE] try_update_reserved_snapshot (ob_tenant_freeze_info_mgr.cpp:793) [4463][T1001_TimerWK3_FreInfoReload][T1001][Y0-0000000000000000-0-0] [lt=21] update reserved snapshot finished(cost_ts=24, reserved_snapshot=0) [2025-05-15 09:40:42.756982] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:42.756999] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:42.757035] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=35][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:42.757049] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:186) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=13] sql audit evict mem start(request_manager_->get_tenant_id()=1001, evict_low_mem_level=0, evict_high_mem_level=8388608, size_used=0, mem_used=12478464) [2025-05-15 09:40:42.757087] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=1, elapse_time=37, size_used=0, mem_used=0) [2025-05-15 09:40:42.757116] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4462][T1001_TimerWK2_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=9] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:42.758279] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3954][TimerWK1_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=15] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44882821939, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:42.762839] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.772947] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.783057] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.793162] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.801949] INFO [STORAGE] operator() (ob_tenant_freezer.cpp:122) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] ====== tenant freeze timer task ====== [2025-05-15 09:40:42.802001] INFO [STORAGE] get_ls_tx_data_memory_info_ (ob_tenant_freezer.cpp:687) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=17] TxData Memory Statistic(logstream info): (ls_id={id:1}, Frozen TxData Memory(MB)=0, Active TxData Memory(MB)=39) [2025-05-15 09:40:42.802027] INFO [STORAGE] check_and_freeze_tx_data_ (ob_tenant_freezer.cpp:593) [4442][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=23] TxData Memory Statistic : (Tenant Total Memory(MB)=4119, Tenant Frozen TxData Memory(MB)=0, Tenant Active TxData Memory(MB)=39, Freeze TxData Trigger Memory(MB)=82, Total TxDataTable Hold Memory(MB)=48, Total TxDataTable Memory Limit(MB)=823) [2025-05-15 09:40:42.803268] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.807675] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.808507] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.808536] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042808500) [2025-05-15 09:40:42.808551] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302042608503, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.808582] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12345056743, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.808599] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.808702] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=102][errcode=-4023] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.808726] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1001}, delta=12345056743, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.808739] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.813371] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.820362] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4498][T1001_LSArchive][T1001][YB427F000001-000635291C8C6112-0-0] [lt=19] gc stale ls task succ [2025-05-15 09:40:42.823475] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.825604] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:42.825628] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4304][T1_LogLoop][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.826894] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4595][T1002_LogLoop][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.827661] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4460][T1001_TimerWK0_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-05-15 09:40:42.827679] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4460][T1001_TimerWK0_CKClogDisk][T1001][Y0-0000000000000000-0-0] [lt=15] get_disk_usage(ret=0, capacity(MB):=1536, used(MB):=1225) [2025-05-15 09:40:42.830206] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.830232] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042830203) [2025-05-15 09:40:42.830249] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302042630146, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.830278] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12345102309, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.830286] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12344929816, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.830295] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.830301] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.833591] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.837574] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4475][T1001_LogLoop][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.843697] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.843763] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.843782] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.843797] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.843820] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.843833] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.843844] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.843857] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.843872] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042843870, replica_locations:[]}) [2025-05-15 09:40:42.843887] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.843909] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.845061] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.845083] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.845098] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.845119] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.845133] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.845144] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.845158] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.845170] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042845170, replica_locations:[]}) [2025-05-15 09:40:42.845225] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=1426456, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=330, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.851139] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=24] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=4) [2025-05-15 09:40:42.853802] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.861830] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4247][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=16] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:42.863905] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.868321] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:243) [4461][T1001_TimerWK1_Flush][T1001][Y0-0000000000000000-0-0] [lt=15] ====== traversal_flush timer task ====== [2025-05-15 09:40:42.868516] INFO [MDS] flush (mds_table_mgr.cpp:184) [4461][T1001_TimerWK1_Flush][T1001][YB427F000001-000635291D5C6447-0-0] [lt=17] [FREEZE]no need do flush cause min_rec_scn is larger than freezing scn(ret=0, ret="OB_SUCCESS", ls_->get_ls_id()={id:1}, recycle_scn={val:4611686018427387903, v:0}, need_freeze=false, order_flusher_for_some={high_priority_flusher:size:32, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, {tablet_id:{id:60157}, rec_scn:{val:1747274416328312026, v:0}}, {tablet_id:{id:403}, rec_scn:{val:1747274416328312027, v:0}}, {tablet_id:{id:101079}, rec_scn:{val:1747274416328312028, v:0}}, {tablet_id:{id:50307}, rec_scn:{val:1747274416328312029, v:0}}, {tablet_id:{id:50272}, rec_scn:{val:1747274416328312030, v:0}}, {tablet_id:{id:383}, rec_scn:{val:1747274416328312031, v:0}}, {tablet_id:{id:238}, rec_scn:{val:1747274416328312032, v:0}}, {tablet_id:{id:60306}, rec_scn:{val:1747274416328312033, v:0}}, {tablet_id:{id:50333}, rec_scn:{val:1747274416328312034, v:0}}, {tablet_id:{id:150}, rec_scn:{val:1747274416328312035, v:0}}, {tablet_id:{id:50238}, rec_scn:{val:1747274416328312036, v:0}}, {tablet_id:{id:406}, rec_scn:{val:1747274416328312037, v:0}}, {tablet_id:{id:50531}, rec_scn:{val:1747274416328312038, v:0}}, {tablet_id:{id:1}, rec_scn:{val:1747274416328312039, v:0}}, {tablet_id:{id:60330}, rec_scn:{val:1747274416328312040, v:0}}, {tablet_id:{id:50319}, rec_scn:{val:1747274416328312041, v:0}}, {tablet_id:{id:50377}, rec_scn:{val:1747274416328312042, v:0}}, {tablet_id:{id:396}, rec_scn:{val:1747274416328312043, v:0}}, {tablet_id:{id:467}, rec_scn:{val:1747274416328312044, v:0}}, {tablet_id:{id:60407}, rec_scn:{val:1747274416328312045, v:0}}, {tablet_id:{id:50430}, rec_scn:{val:1747274416328312046, v:0}}, {tablet_id:{id:50104}, rec_scn:{val:1747274416328312047, v:0}}, {tablet_id:{id:50425}, rec_scn:{val:1747274416328312048, v:0}}, {tablet_id:{id:50480}, rec_scn:{val:1747274416328312049, v:0}}, {tablet_id:{id:60503}, rec_scn:{val:1747274416328312050, v:0}}, {tablet_id:{id:60390}, rec_scn:{val:1747274416328312051, v:0}}, {tablet_id:{id:50436}, rec_scn:{val:1747274416328312052, v:0}}, {tablet_id:{id:50160}, rec_scn:{val:1747274416328312053, v:0}}, {tablet_id:{id:474}, rec_scn:{val:1747274416328312054, v:0}}, {tablet_id:{id:60393}, rec_scn:{val:1747274416328312055, v:0}}, {tablet_id:{id:50451}, rec_scn:{val:1747274416328312056, v:0}}, }, max_consequent_callbacked_scn={val:1747289697900328000, v:0}, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:42.868675] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:278) [4461][T1001_TimerWK1_Flush][T1001][YB427F000001-000635291D5C6447-0-0] [lt=153] succeed to traversal_flush(ret=0, ls_cnt=1) [2025-05-15 09:40:42.874013] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.884119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.890168] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=52][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4023, dropped:32, tid:4630}]) [2025-05-15 09:40:42.894222] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.904326] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.907732] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:42.908580] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12345157208, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.908611] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.908667] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=54][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.908684] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.908695] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.908711] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12345157208, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:42.908725] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.908737] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:42.908749] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:42.908761] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:42.908773] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.908783] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:42.908794] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042908563) [2025-05-15 09:40:42.914429] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.915298] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4462][T1001_TimerWK2_LogUpdater][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.924539] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.930273] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=0] get wrs ts(ls_id={id:1}, delta=12345201727, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:42.930297] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.930319] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:42.930335] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.930344] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.930333] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=0][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:42.930355] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.930357] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302042930327) [2025-05-15 09:40:42.930366] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:42.930371] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302042730276, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:42.930374] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042930261) [2025-05-15 09:40:42.930393] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12345029234, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:42.930408] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:42.930425] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:42.930438] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:42.930448] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:42.930461] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.930470] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:42.930477] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302042930385) [2025-05-15 09:40:42.934132] INFO [PALF] next (palf_iterator.h:183) [4666][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:17, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:32, read_io_size:62517248, read_disk_cost_ts:874359, start_lsn:{lsn:872361984}}}) [2025-05-15 09:40:42.934139] INFO [PALF] next (palf_iterator.h:183) [4663][T1002_ReplaySrv][T1002][Y0-0000000000000000-0-0] [lt=29] [PALF STAT ITERATOR INFO](io_ctx={user:"REPLAY", palf_id:1001, iterator_info:{allow_filling_cache:true, hot_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, cold_cache_stat:{hit_cnt:0, miss_cnt:3, cache_read_size:0, hit_ratio:"0.000000000000000000e+00"}, read_io_cnt:4, read_io_size:9224192, read_disk_cost_ts:121281, start_lsn:{lsn:738152448}}}) [2025-05-15 09:40:42.934644] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.937627] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:503) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=13] try inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4850660378, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:42.937733] INFO [LIB] do_thread_recycle (threads.cpp:158) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=30] recycle one thread(this=0x7f1ae73fe3b0, total=5, remain=4) [2025-05-15 09:40:42.937748] INFO [COMMON] try_inc_thread_count (ob_dynamic_thread_pool.cpp:508) [3958][qth_mgr][T0][Y0-0000000000000000-0-0] [lt=15] inc thread count(*this={name:TimerWK, this:0x7f1ae73fe3b0, min_thread_cnt:4, max_thread_cnt:128, running_thread_cnt:1, threads_idle_time:4850660378, tenant_id:1}, cur_thread_count=5, cnt=-1, new_thread_count=4) [2025-05-15 09:40:42.938081] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.938105] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.938123] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.938137] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.938157] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.938171] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.938183] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.938197] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.938209] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:347) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10][errcode=-4638] no leader finded(ret=-4638, ret="OB_RS_NOT_MASTER", leader_exist=false, ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.938222] INFO [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:358) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13] [RS_MGR] new master rootserver found(rootservice="0.0.0.0:0", cluster_id=1747291386) [2025-05-15 09:40:42.938235] WDIAG [SHARE] renew_master_rootserver (ob_rs_mgr.cpp:303) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=12][errcode=-4638] failed to renew master rootserver(ret=-4638, ret="OB_RS_NOT_MASTER") [2025-05-15 09:40:42.938248] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:163) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] renew_master_rootserver failed(tmp_ret=-4638) [2025-05-15 09:40:42.938294] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=19][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.938307] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.938320] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.938337] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.938349] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.938359] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.938370] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.938383] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042938382, replica_locations:[]}) [2025-05-15 09:40:42.938403] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:165) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=0][errcode=-4122] fail to refresh core partition(tmp_ret=-4721) [2025-05-15 09:40:42.938454] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=8][errcode=-4122] check_blacklist failed(ret=-4122) [2025-05-15 09:40:42.938469] WDIAG [SHARE.PT] update (ob_rpc_ls_table.cpp:185) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=9][errcode=-4122] report sys_tenant's ls through rpc failed(replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, rs_addr="172.19.0.2:2882", ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.938520] INFO [SHARE.PT] update (ob_rpc_ls_table.cpp:188) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=52] update sys_tenant's ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:42.938541] WDIAG [SHARE.PT] update (ob_ls_table_operator.cpp:141) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=20][errcode=-4122] update replica failed(ret=-4122, ret="OB_RPC_POST_ERROR", replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:42.938562] INFO [SHARE.PT] update (ob_ls_table_operator.cpp:145) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=20] update ls replica(ret=-4122, ret="OB_RPC_POST_ERROR", time_used=130, replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}, inner_table_only=false) [2025-05-15 09:40:42.938584] WDIAG [SERVER] try_report_sys_ls (ob_lease_state_mgr.cpp:221) [3947][observer][T1][YB427F000001-000635291BCC6115-0-0] [lt=21][errcode=-4122] fail to report sys log stream(ret=-4122, ret="OB_RPC_POST_ERROR", ls_replica={modify_time_us:0, create_time_us:0, tenant_id:1, ls_id:{id:1}, server:"127.0.0.1:2882", sql_port:2881, role:2, member_list:[{server:"172.19.0.2:2882", timestamp:1}], replica_type:0, proposal_id:4, replica_status:"NORMAL", restore_status:{status:0}, property:{memstore_percent_:100}, unit_id:1, zone:"zone1", paxos_replica_number:1, data_size:0, required_size:0, in_member_list:false, member_time_us:0, learner_list:{learner_num:0, learner_array:[]}, in_learner_list:false, rebuild:false}) [2025-05-15 09:40:42.938606] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:152) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=20][errcode=-4122] fail to try report sys log stream [2025-05-15 09:40:42.938620] WDIAG [SERVER] register_self_busy_wait (ob_lease_state_mgr.cpp:159) [3947][observer][T0][YB427F000001-000635291BCC6115-0-0] [lt=13][errcode=-4122] register failed, will try again(ret=-4122, ret="OB_RPC_POST_ERROR", retry latency=2) [2025-05-15 09:40:42.943570] INFO eloop_run (eloop.c:145) [4019][pnio1][T0][Y0-0000000000000000-0-0] [lt=24] PNIO [ratelimit] time: 1747302042943562, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007613, add_bytes: 0 [2025-05-15 09:40:42.944761] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.945336] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.945355] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.945369] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.945393] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.945406] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.945417] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.945429] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.945442] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042945441, replica_locations:[]}) [2025-05-15 09:40:42.945457] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:42.945480] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:42.946719] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:42.946741] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.946756] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:42.946777] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.946789] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:42.946799] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:42.946812] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:42.946824] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302042946823, replica_locations:[]}) [2025-05-15 09:40:42.946884] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=1324798, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=331, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:42.949445] INFO eloop_run (eloop.c:145) [4023][pnio2][T0][Y0-0000000000000000-0-0] [lt=20] PNIO [ratelimit] time: 1747302042949443, bytes: 0, bw: 0.000000 MB/s, add_ts: 1007616, add_bytes: 0 [2025-05-15 09:40:42.954874] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.958574] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=43] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44880724787, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:42.959823] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=69] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:42.959851] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=22] temporary file module use no blocks [2025-05-15 09:40:42.959861] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=8] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:42.959882] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=18] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=5313, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:42.959901] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4485][T1001_TFSwap][T1001][Y0-0000000000000000-0-0] [lt=19] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11805850235, flush_io_finished_ret_:0, flush_io_finished_round_:1517, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:42.964788] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=21] table not exist(tenant_id=1, database_id=201001, table_name=__all_server, table_name.ptr()="data_size:12, data:5F5F616C6C5F736572766572", ret=-5019) [2025-05-15 09:40:42.964816] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=24][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_server, ret=-5019) [2025-05-15 09:40:42.964828] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_server, db_name=oceanbase) [2025-05-15 09:40:42.964843] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=14][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_server) [2025-05-15 09:40:42.964935] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=90][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:42.964947] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:42.964962] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_server' doesn't exist [2025-05-15 09:40:42.964969] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=6][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:42.964978] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=7][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:42.964982] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.964988] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=10][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:42.965000] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:42.965010] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=10][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:42.965020] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:42.965030] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=9][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:42.965049] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=7][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:42.965062] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=12][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:42.965076] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=12][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:42.965088] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:42.965099] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, ret=-5019) [2025-05-15 09:40:42.965114] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:42.965126] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=11][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:42.965148] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=17][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:42.965171] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:42.965178] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=7][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:42.965187] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:42.965207] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=11][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:42.965221] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4535][T1002_Occam][T1][YB427F000001-000635291FAC6241-0-0] [lt=13][errcode=0] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:42.965235] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4535][T1002_Occam][T1002][YB427F000001-000635291FAC6241-0-0] [lt=12][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882"}, aret=-5019, ret=-5019) [2025-05-15 09:40:42.965248] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:42.965261] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:42.965273] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:42.965284] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cb0f0, start=1747302042964547, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:42.965298] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:42.965309] WDIAG get_my_sql_result_ (ob_table_access_helper.h:501) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5019] GCTX.sql_proxy_ read failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, tenant_id=1, columns=0x7f1a84e563f8, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882, sql=SELECT zone FROM __all_server where svr_ip='127.0.0.1' and svr_port=2882, columns_str="zone") [2025-05-15 09:40:42.965328] WDIAG read_and_convert_to_values_ (ob_table_access_helper.h:388) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5019] fail to get ObMySQLResult(ret=-5019, ret="OB_TABLE_NOT_EXIST", MTL_ID()=1002, table=__all_server, condition=where svr_ip='127.0.0.1' and svr_port=2882) [2025-05-15 09:40:42.965428] WDIAG [COORDINATOR] get_self_zone_name (table_accessor.cpp:629) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5019] get zone from __all_server failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", columns=0x7f1a84e563f8, where_condition="where svr_ip='127.0.0.1' and svr_port=2882", zone_name_holder=) [2025-05-15 09:40:42.965446] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:562) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5019] get self zone name failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:42.965460] WDIAG [COORDINATOR] get_all_ls_election_reference_info (table_accessor.cpp:571) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5019] zone name is empty(ret=-5019, ret="OB_TABLE_NOT_EXIST", all_ls_election_reference_info=[]) [2025-05-15 09:40:42.965468] WDIAG [COORDINATOR] refresh (ob_leader_coordinator.cpp:137) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] get all ls election reference info failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:42.965497] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4535][T1002_Occam][T1002][Y0-0000000000000000-0-0] [lt=10] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:42.975096] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.985207] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.990399] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=29][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:0, dropped:45, tid:3961}, {errcode:-5627, dropped:48, tid:4602}]) [2025-05-15 09:40:42.995312] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:42.996074] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEC-0-0] [lt=12][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.996096] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEC-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.996120] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEC-0-0] [lt=11][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.996138] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CEC-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.996153] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CEC-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 "}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.996169] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:42.996184] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.996198] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.996212] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302042996047, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:42.996226] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.996241] WDIAG [SHARE] load_pure_tenant_info_ (ob_tenant_info_proxy.cpp:430) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to read(ret=-5627, ret="OB_SCHEMA_EAGAIN", exec_tenant_id=1001, sql=select ORA_ROWSCN, * from __all_tenant_info where tenant_id = 1002 ) [2025-05-15 09:40:42.996321] WDIAG [SHARE] load_tenant_info (ob_tenant_info_proxy.cpp:385) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5627] failed to load purge tenant info(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, for_update=false) [2025-05-15 09:40:42.997580] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CED-0-0] [lt=20][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.997597] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CED-0-0] [lt=17][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:42.997615] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CED-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:42.997627] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4542][T1002_TenantInf][T1001][YB427F000001-000635291DEC6CED-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:42.997638] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4542][T1002_TenantInf][T1002][YB427F000001-000635291DEC6CED-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create"}, aret=-5627, ret=-5627) [2025-05-15 09:40:42.997648] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:42.997658] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:42.997667] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:42.997676] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca700, start=1747302042997570, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:42.997686] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:42.997693] WDIAG [SHARE] select_service_name_sql_helper_ (ob_service_name_proxy.cpp:294) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=5][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:42.997732] WDIAG [SHARE] select_all_service_names_with_epoch (ob_service_name_proxy.cpp:250) [4542][T1002_TenantInf][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to execute select_service_name_sql_helper_(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT s.*, e.value as epoch FROM __all_service AS s RIGHT JOIN __all_service_epoch AS e ON s.tenant_id = e.tenant_id WHERE e.tenant_id = 1002 and e.name='service_name_epoch' ORDER BY s.gmt_create) [2025-05-15 09:40:42.999036] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4631][T1002_LSArchive][T1002][YB427F000001-000635291DFC6112-0-0] [lt=16] gc stale ls task succ [2025-05-15 09:40:43.005419] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.007793] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:43.007829] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4580][T1002_TimerWK3_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:43.008199] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.008214] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.008226] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.008239] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.008251] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:43.008264] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:43.008281] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:43.008296] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=13] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=1, size_used=0, mem_used=8318976) [2025-05-15 09:40:43.008336] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4577][T1002_TimerWK0_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:43.008620] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.008644] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043008614) [2025-05-15 09:40:43.008660] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302042808564, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.008696] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12345256613, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.008714] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.008737] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.008754] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.008762] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.008780] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1001}, delta=12345256613, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.008793] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.008807] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.008816] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:43.008826] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:43.008837] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.008848] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.008858] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043008674) [2025-05-15 09:40:43.015528] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.020644] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=0][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.020663] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:43.020680] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:43.020696] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.020710] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:43.020723] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.020754] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=31][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.020762] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:43.020775] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb440, start=1747302043020624, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.020789] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:43.020796] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=6][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.020850] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:43.020860] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:43.020872] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:43.020883] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:43.024903] INFO [STORAGE.COMPACTION] runTimerTask (ob_tenant_tablet_scheduler_task_mgr.cpp:164) [4463][T1001_TimerWK3_MediumLoop][T1001][YB427F000001-000635291F5C6473-0-0] [lt=7] MediumCheckTask(cost_ts=7) [2025-05-15 09:40:43.025637] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.030322] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.030356] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043030318) [2025-05-15 09:40:43.030374] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302042830264, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.030395] INFO [STORAGE.TRANS] self_check (ob_tenant_weak_read_cluster_service.cpp:826) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [WRS] [TENANT_WEAK_READ_SERVICE] [CLUSTER_SERVICE] [SELF_CHECK] done(ret=0, ret="OB_SUCCESS", tenant_id=1, need_start_service=false, need_stop_service=false, need_change_leader=false, is_in_service()=false, can_update_version=false, cur_leader_epoch=0, start_service_tstamp_=0, error_count_for_change_leader_=0, last_error_tstamp_for_change_leader_=0) [2025-05-15 09:40:43.030399] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21] get wrs ts(ls_id={id:1}, delta=12345129679, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:43.030416] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.030432] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:43.030434] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=28] get wrs ts(ls_id={id:1}, delta=12345302172, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:43.030449] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.030457] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.030463] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:43.030465] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.030473] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.030476] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.030480] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.030483] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.030499] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043030386) [2025-05-15 09:40:43.030501] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.030508] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.030516] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043030422) [2025-05-15 09:40:43.035743] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.036195] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:37) [4462][T1001_TimerWK2_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=1] ====== [emptytablet] empty shell timer task ======(GC_EMPTY_TABLET_SHELL_INTERVAL=5000000) [2025-05-15 09:40:43.036224] INFO [STORAGE] runTimerTask (ob_empty_shell_task.cpp:103) [4462][T1001_TimerWK2_TabletShell][T1001][Y0-0000000000000000-0-0] [lt=15] [emptytablet] succeed to change tablet to empty shell(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=81) [2025-05-15 09:40:43.045857] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.047016] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.047040] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.047078] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=38][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.047093] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.047106] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.047121] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.047139] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.047155] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.047172] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.047180] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.047194] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.047210] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043047208, replica_locations:[]}) [2025-05-15 09:40:43.047228] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:43.047254] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:43.047266] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:43.047290] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.047339] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11806937815, cache_obj->added_lc()=false, cache_obj->get_object_id()=15854, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.048583] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.048604] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.048620] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.048633] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.048641] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.048655] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.048670] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.048684] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.048699] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.048710] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.048724] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.048737] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043048736, replica_locations:[]}) [2025-05-15 09:40:43.048803] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=1222879, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=332, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:43.051230] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=19] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=3) [2025-05-15 09:40:43.055961] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.066063] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.070986] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:43.071023] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=27][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.071057] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=34][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:43.071070] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:43.071080] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:43.071160] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4504][T1001_HAService][T1001][Y0-0000000000000000-0-0] [lt=80][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:43.076169] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.076297] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:43.076339] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:43.076336] WDIAG [SHARE.LOCATION] batch_process_tasks (ob_ls_location_service.cpp:540) [4065][SysLocAsyncUp0][T0][YB427F000001-000635291AFC96C4-0-0] [lt=25][errcode=0] tenant schema is not ready, need wait(ret=0, ret="OB_SUCCESS", superior_tenant_id=1, task={cluster_id:1747291386, tenant_id:1, ls_id:{id:1}, renew_for_tenant:false, add_timestamp:1747302043076314}) [2025-05-15 09:40:43.076370] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:43.076389] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:43.076405] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:43.076763] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.076775] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=1][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:43.076791] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:43.084411] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:147) [4463][T1001_TimerWK3_][T1001][Y0-0000000000000000-0-0] [lt=17] ====== [tabletchange] timer task ======(GC_CHECK_INTERVAL=5000000) [2025-05-15 09:40:43.084436] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:181) [4463][T1001_TimerWK3_][T1001][Y0-0000000000000000-0-0] [lt=17] [tabletchange] task check ls(ls->get_ls_id()={id:1}, tablet_persist_trigger=0) [2025-05-15 09:40:43.084454] INFO [STORAGE] runTimerTask (ob_tablet_gc_service.cpp:275) [4463][T1001_TimerWK3_][T1001][Y0-0000000000000000-0-0] [lt=16] [tabletchange] succeed to gc_tablet(ret=0, ret="OB_SUCCESS", ls_cnt=1, times=87) [2025-05-15 09:40:43.086277] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.093864] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553308, support_ssl=false) [2025-05-15 09:40:43.094049] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=74] sm conn init succ(conn.sessid_=3221553308, sess.client_addr_="172.22.0.1:40526") [2025-05-15 09:40:43.094071] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553308, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.094226] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=26] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.094268] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.094298] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.094318] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.094330] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.094338] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.094360] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.094378] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553308, proxy_sessid=0) [2025-05-15 09:40:43.094416] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=10][errcode=0] server close connection(sessid=3221553308, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.094427] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553308, proxy_sessid=0) [2025-05-15 09:40:43.094436] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:43.094449] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B5-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553308, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.094510] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553308, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302043093374, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.094535] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22] kill and revert session(conn.sessid_=3221553308, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.094547] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553308, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302043093374, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.094569] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553308) [2025-05-15 09:40:43.094587] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553308, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.096385] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.106497] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.107853] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:43.108694] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12345356970, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.108720] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.108741] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.108757] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.108767] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.108783] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12345356970, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.108795] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=1, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.108809] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.108820] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:43.108829] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:43.108840] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.108853] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.108868] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043108673) [2025-05-15 09:40:43.110298] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:122) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=15] ====== checkpoint timer task ====== [2025-05-15 09:40:43.110926] INFO [STORAGE.TRANS] get_rec_scn (ob_trans_ctx_mgr_v4.cpp:1417) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=12] succ to get rec scn(*this={this:0x7f1a94c04030, ls_id:{id:1}, tenant_id:1001, tx_ls_state_mgr:{ls_id_:{id:1}, cur_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, prev_state_.state_val_:{state_:"F_WORKING", offline_flag_:0, block_start_tx_flag_:0, block_start_normal_tx_flag_:0, block_start_readonly_flag_:0}, max_applying_start_working_ts_:{val:18446744073709551615, v:3}, max_applied_start_working_ts_:{val:18446744073709551615, v:3}}, total_tx_ctx_count:0, active_tx_count:0, ls_retain_ctx_mgr:{retain_ctx_list_.get_size():0, max_wait_ckpt_ts_:{val:18446744073709551615, v:3}, last_push_gc_task_ts_:1747300521088740, skip_remove_cnt_:0}, aggre_rec_scn:{val:18446744073709551615, v:3}, prev_aggre_rec_scn:{val:18446744073709551615, v:3}, uref:1007}, aggre_rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:43.110965] INFO [STORAGE.TRANS] get_rec_scn (ob_tx_ctx_memtable.cpp:178) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=38] tx ctx memtable get rec scn(this={ObITable:{this:0x7f1a95378080, key:{tablet_id:{id:49401}, column_group_idx:0, slice_range:{start_slice_idx:0, end_slice_idx:0}, table_type:"TX_CTX_MEMTABLE", scn_range:{start_scn:{val:1, v:0}, end_scn:{val:2, v:0}}}, ref_cnt:2, upper_trans_version:-4007, timestamp:0}, this:0x7f1a95378080, snapshot_version:{val:4611686018427387903, v:0}, ls_id:{id:1}, is_frozen:false}, rec_scn={val:4611686018427387903, v:0}) [2025-05-15 09:40:43.111141] INFO [MDS] get_rec_scn (mds_table_mgr.cpp:254) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=37] [FREEZE]get rec_scn from MdsTableMgr(ret=0, ret="OB_SUCCESS", min_rec_scn={val:1747274416328312025, v:0}, order_flusher_for_one={high_priority_flusher:size:1, ordered_tablets:{{tablet_id:{id:518}, rec_scn:{val:1747274416328312025, v:0}}, }, *this={this:0x7f1a979cd150, is_inited:true, freezing_scn:{val:0, v:0}, ls:{running_state:{state:1}, ls_meta:{tenant_id:1001, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274408663046000, v:0}, clog_base_lsn:{lsn:5167067136}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:18446744073709551615, v:3}, tablet_change_checkpoint_scn:{val:1747274416328312024, v:0}, all_id_meta:{id_meta:[{limited_id:1747289712065764000, latest_log_ts:{val:1747289695412573001, v:0}}, {limited_id:14000001, latest_log_ts:{val:1747286011294311001, v:0}}, {limited_id:40000001, latest_log_ts:{val:1747186182975367008, v:0}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1aaf9f6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}}) [2025-05-15 09:40:43.111229] INFO [STORAGE.TRANS] get_rec_scn (ob_ls_tx_service.cpp:627) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=86] [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="TX_DATA_MEMTABLE_TYPE", common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={ObIMemtableMgr:{Memtables:this:0x7f1a979cc1b0, ref_cnt:1, is_inited:true, tablet_id:{id:49402}, freezer:0x7f1a979d7010, memtable_head:0, memtable_tail:1, t3m:0x7f1aaa0ea030, tables:[0x7f1a952dc080, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null]}, is_freezing:false, ls_id:{id:1}, mini_merge_recycle_commit_versions_ts:0, tx_data_table:0x7f1a979d8d10, ls_tablet_svr:0x7f1a979cc190}, min_rec_scn={val:1747274408663046000, v:0}, ls_id_={id:1}) [2025-05-15 09:40:43.111263] INFO [STORAGE] get_rec_scn (ob_ls_ddl_log_handler.cpp:493) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=26] [CHECKPOINT] ObLSDDLLogHandler::get_rec_scn(ret=0, ls_id={id:1}, barrier_tablet_id={id:0}, rec_scn={val:4611686018427387903, v:0}, last_rec_scn={val:1, v:0}) [2025-05-15 09:40:43.111279] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=13] get rec log scn(service_type_=0, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:43.111290] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=11] get rec log scn(service_type_=1, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:43.111304] INFO [STORAGE.TRANS] get_rec_scn (ob_id_service.cpp:299) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=12] get rec log scn(service_type_=2, rec_log_ts={val:4611686018427387903, v:0}) [2025-05-15 09:40:43.111319] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=9] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:43.111340] INFO [STORAGE.TABLELOCK] get_rec_scn (ob_lock_memtable.cpp:1207) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=17] rec_scn of ObLockMemtable is (rec_scn_={val:4611686018427387903, v:0}, flushed_scn_={val:0, v:0}, pre_rec_scn_={val:4611686018427387903, v:0}, freeze_scn_={val:0, v:0}, max_committed_scn_={val:18446744073709551615, v:3}, is_frozen_=false, ls_id_={id:1}) [2025-05-15 09:40:43.111355] INFO [STORAGE] update_clog_checkpoint (ob_checkpoint_executor.cpp:185) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=13] [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1747274408663046000, v:0}, checkpoint_scn_in_ls_meta={val:1747274408663046000, v:0}, ls_id={id:1}, service_type="TRANS_SERVICE") [2025-05-15 09:40:43.111377] WDIAG [PALF] set_base_lsn (palf_handle_impl.cpp:1672) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=0] no need to set new base lsn, curr base lsn is greater than or equal to new base lsn(this={palf_id:1, self:"127.0.0.1:2882", has_set_deleted:false}, log_snapshot_meta={version:2, base_lsn:{lsn:5167067136}, prev_log_info:{log_id:0, lsn:{lsn:0}, scn:{val:1, v:0}, log_proposal_id:9223372036854775807, accum_checksum:-1}, prev_log_tail_lsn:{lsn:0}}, new_base_lsn={lsn:5167067136}, lsn={lsn:5167067136}) [2025-05-15 09:40:43.111402] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:179) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=24] [CHECKPOINT] advance palf base lsn successfully(checkpoint_lsn={lsn:5167067136}, ls->get_ls_id()={id:1}) [2025-05-15 09:40:43.111414] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:186) [4463][T1001_TimerWK3_TxCkpt][T1001][Y0-0000000000000000-0-0] [lt=9] succeed to update_clog_checkpoint(ret=0, ls_cnt=1) [2025-05-15 09:40:43.116200] WDIAG [STORAGE] get_tenant_status_ (ob_tenant_snapshot_service.cpp:216) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:43.116220] WDIAG [STORAGE] decide_running_mode_ (ob_tenant_snapshot_service.cpp:327) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-5157] fail to get_tenant_status_(ret=-5157, ret="OB_TENANT_NOT_EXIST") [2025-05-15 09:40:43.116233] INFO [STORAGE] run1 (ob_tenant_snapshot_service.cpp:745) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=12] fail to decide_running_mode_(ret=-5157, ret="OB_TENANT_NOT_EXIST", this={is_inited_:true, is_running_:true, running_mode_:0, meta_loaded_:false, tg_id_:460}) [2025-05-15 09:40:43.116250] INFO [STORAGE] calculate_idle_time_ (ob_tenant_snapshot_service.cpp:824) [4644][T1002_TSnapSvc][T1002][Y0-0000000000000000-0-0] [lt=16] ObTenantSnapshotService thread idle time(tg_id_=460, idle_time=1000, running_mode_=0) [2025-05-15 09:40:43.116604] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.120566] INFO [ARCHIVE] gc_stale_ls_task_ (ob_ls_mgr.cpp:572) [4333][T1_LSArchiveMgr][T1][YB427F000001-000635291B6C6112-0-0] [lt=21] gc stale ls task succ [2025-05-15 09:40:43.120738] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.120759] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=21][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:43.120800] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=29][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:43.120817] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.120830] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:43.120843] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.120852] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.120866] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:43.120882] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca060, start=1747302043120711, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.120910] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=27][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:43.120924] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.120996] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:43.121008] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:43.121018] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:43.121030] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:43.126712] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.129362] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=29] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:43.129389] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-05-15 09:40:43.129401] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=10] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:43.129420] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=16] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=47564, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:43.129435] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4605][T1002_TFSwap][T1002][Y0-0000000000000000-0-0] [lt=15] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11806019786, flush_io_finished_ret_:0, flush_io_finished_round_:1517, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:1, normal_idle_loop_cnt_:1, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:43.130394] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8] get wrs ts(ls_id={id:1}, delta=12345402457, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:43.130417] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.130440] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=21][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:43.130455] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.130465] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.130456] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.130480] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.130503] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.130484] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=25][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043130446) [2025-05-15 09:40:43.130509] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043130376) [2025-05-15 09:40:43.130514] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302042930385, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.130548] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=12345229964, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:43.130566] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.130588] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:43.130607] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.130620] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.130637] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.130649] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.130661] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043130534) [2025-05-15 09:40:43.136819] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.138675] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=29] new mysql sessid created(conn.sessid_=3221553310, support_ssl=false) [2025-05-15 09:40:43.138799] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=23] sm conn init succ(conn.sessid_=3221553310, sess.client_addr_="172.22.0.1:40534") [2025-05-15 09:40:43.138823] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a9d9fc030, session_id:3221553310, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.138851] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553311, support_ssl=false) [2025-05-15 09:40:43.138889] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553311, sess.client_addr_="172.22.0.1:40536") [2025-05-15 09:40:43.138902] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a9d9fcdb0, session_id:3221553311, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.138914] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553312, support_ssl=false) [2025-05-15 09:40:43.138988] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=39] sm conn init succ(conn.sessid_=3221553312, sess.client_addr_="172.22.0.1:40540") [2025-05-15 09:40:43.139007] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.139011] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] accept one succ(*s={this:0x7f1a1de04030, session_id:3221553312, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:147}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.139042] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.139073] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.139099] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=23][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.139102] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.139105] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221553313, support_ssl=false) [2025-05-15 09:40:43.139116] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.139126] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=14] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.139132] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.139140] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553313, sess.client_addr_="172.22.0.1:40552") [2025-05-15 09:40:43.139146] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=19][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.139152] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553313, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.139164] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.139165] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139175] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.139186] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.139185] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=16][errcode=-4018] get session fail(ret=-4018, sessid=3221553310, proxy_sessid=0) [2025-05-15 09:40:43.139195] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=18] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.139206] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139221] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553311, proxy_sessid=0) [2025-05-15 09:40:43.139221] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=14][errcode=0] server close connection(sessid=3221553310, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139233] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553310, proxy_sessid=0) [2025-05-15 09:40:43.139231] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.139241] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:43.139247] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=13][errcode=0] server close connection(sessid=3221553311, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139258] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553311, proxy_sessid=0) [2025-05-15 09:40:43.139250] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A6-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553310, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.139265] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=6][errcode=-4016] session is null [2025-05-15 09:40:43.139259] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.139272] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E6-0-0] [lt=5] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553311, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.139279] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.139290] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=10][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.139303] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.139328] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=12] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139319] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fc030, session_id:3221553310, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.139343] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553312, proxy_sessid=0) [2025-05-15 09:40:43.139273] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.139356] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4015] close sql sock by user req(*s={this:0x7f1a9d9fcdb0, session_id:3221553311, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.139373] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] kill and revert session(conn.sessid_=3221553310, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.139378] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=9][errcode=0] server close connection(sessid=3221553312, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139383] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] kill and revert session(conn.sessid_=3221553311, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.139388] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553312, proxy_sessid=0) [2025-05-15 09:40:43.139395] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:43.139393] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=9] can close safely, do destroy(*s={this:0x7f1a9d9fc030, session_id:3221553310, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:229, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.139394] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=98] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.139407] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] mark session id unused(conn.sessid_=3221553310) [2025-05-15 09:40:43.139406] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=20][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.139407] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B6-0-0] [lt=9] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553312, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.139418] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.139430] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.139444] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=13][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.139429] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] connection close(sessid=3221553310, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.139456] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139457] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=27] can close safely, do destroy(*s={this:0x7f1a9d9fcdb0, session_id:3221553311, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:227, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.139466] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553313, proxy_sessid=0) [2025-05-15 09:40:43.139472] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=14] mark session id unused(conn.sessid_=3221553311) [2025-05-15 09:40:43.139497] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=10][errcode=0] server close connection(sessid=3221553313, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139471] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221553312, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.139482] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553311, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.139511] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=13][errcode=-4018] get session fail(ret=-4018, sessid=3221553313, proxy_sessid=0) [2025-05-15 09:40:43.139519] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:43.139517] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=41] kill and revert session(conn.sessid_=3221553312, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.139525] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A7-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553313, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.139528] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=11] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221553312, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:147}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.139549] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=19] mark session id unused(conn.sessid_=3221553312) [2025-05-15 09:40:43.139567] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553312, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.139618] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=24][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de1e030, session_id:3221553313, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.139637] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553314, support_ssl=false) [2025-05-15 09:40:43.139642] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553313, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.139678] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11] sm conn init succ(conn.sessid_=3221553314, sess.client_addr_="172.22.0.1:40562") [2025-05-15 09:40:43.139674] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de1e030, session_id:3221553313, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302043138718, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.139691] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16] mark session id unused(conn.sessid_=3221553313) [2025-05-15 09:40:43.139692] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221553314, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.139706] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=8] connection close(sessid=3221553313, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.139870] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=14] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.139894] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.139911] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=16][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.139930] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=18][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.139938] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=7][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.139948] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.139960] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.139974] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553314, proxy_sessid=0) [2025-05-15 09:40:43.139996] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=12][errcode=0] server close connection(sessid=3221553314, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.140008] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553314, proxy_sessid=0) [2025-05-15 09:40:43.140015] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:43.140035] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E7-0-0] [lt=20] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553314, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.140087] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=11][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221553314, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302043139777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.140109] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=19] kill and revert session(conn.sessid_=3221553314, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.140116] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221553314, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302043139777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.140131] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553314) [2025-05-15 09:40:43.140143] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=7] connection close(sessid=3221553314, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.140399] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] new mysql sessid created(conn.sessid_=3221553315, support_ssl=false) [2025-05-15 09:40:43.140435] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] sm conn init succ(conn.sessid_=3221553315, sess.client_addr_="172.22.0.1:40568") [2025-05-15 09:40:43.140449] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=12] accept one succ(*s={this:0x7f1a1de04030, session_id:3221553315, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:144}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.140598] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.140625] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4131][sql_nio3][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.140636] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.140648] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=12][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.140657] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=9][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.140665] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=8][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.140679] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=9] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.140690] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553315, proxy_sessid=0) [2025-05-15 09:40:43.140710] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=11][errcode=0] server close connection(sessid=3221553315, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.140720] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553315, proxy_sessid=0) [2025-05-15 09:40:43.140730] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:43.140738] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B7-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553315, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.140777] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04030, session_id:3221553315, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302043139777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.140795] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553315, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.140805] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=10] can close safely, do destroy(*s={this:0x7f1a1de04030, session_id:3221553315, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:144}, err:5, last_decode_time:1747302043139777, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:228, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.140820] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553315) [2025-05-15 09:40:43.140833] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4131][sql_nio3][T0][Y0-0000000000000000-0-0] [lt=9] connection close(sessid=3221553315, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.141509] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=22] new mysql sessid created(conn.sessid_=3221553316, support_ssl=false) [2025-05-15 09:40:43.141555] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553316, sess.client_addr_="172.22.0.1:40580") [2025-05-15 09:40:43.141571] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] accept one succ(*s={this:0x7f1a1de04db0, session_id:3221553316, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:139}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.141707] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=30] new mysql sessid created(conn.sessid_=3221553317, support_ssl=false) [2025-05-15 09:40:43.141750] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=12] sm conn init succ(conn.sessid_=3221553317, sess.client_addr_="172.22.0.1:40594") [2025-05-15 09:40:43.141753] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.141764] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1acd9fabd0, session_id:3221553317, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:137}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.141779] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.141809] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=18][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.141838] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=28][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.141857] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=19][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.141872] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=15][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.141874] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] new mysql sessid created(conn.sessid_=3221553318, support_ssl=false) [2025-05-15 09:40:43.141899] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=18] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.141916] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=15][errcode=-4018] get session fail(ret=-4018, sessid=3221553316, proxy_sessid=0) [2025-05-15 09:40:43.141924] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=22] sm conn init succ(conn.sessid_=3221553318, sess.client_addr_="172.22.0.1:40602") [2025-05-15 09:40:43.141942] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=13][errcode=0] server close connection(sessid=3221553316, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.141951] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553316, proxy_sessid=0) [2025-05-15 09:40:43.141958] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=7][errcode=-4016] session is null [2025-05-15 09:40:43.141948] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a6f616db0, session_id:3221553318, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.141962] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=16] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.141965] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A8-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553316, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.141992] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4129][sql_nio1][T1][Y0-0000000000000000-0-0] [lt=16] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.141995] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.142004] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=8][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.142016] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.142022] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.142015] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4015] close sql sock by user req(*s={this:0x7f1a1de04db0, session_id:3221553316, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302043140836, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.142034] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=7] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142045] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=10][errcode=-4018] get session fail(ret=-4018, sessid=3221553317, proxy_sessid=0) [2025-05-15 09:40:43.142047] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221553316, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.142060] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=8][errcode=0] server close connection(sessid=3221553317, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142070] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553317, proxy_sessid=0) [2025-05-15 09:40:43.142062] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=14] can close safely, do destroy(*s={this:0x7f1a1de04db0, session_id:3221553316, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:139}, err:5, last_decode_time:1747302043140836, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.142079] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=8][errcode=-4016] session is null [2025-05-15 09:40:43.142084] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=21] mark session id unused(conn.sessid_=3221553316) [2025-05-15 09:40:43.142096] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90A9-0-0] [lt=16] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553317, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.142105] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] connection close(sessid=3221553316, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.142120] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.142144] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4015] close sql sock by user req(*s={this:0x7f1acd9fabd0, session_id:3221553317, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.142166] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=21] kill and revert session(conn.sessid_=3221553317, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.142165] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=21] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.142174] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1acd9fabd0, session_id:3221553317, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:137}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.142189] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553317) [2025-05-15 09:40:43.142186] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=15][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.142201] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=15][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.142200] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4129][sql_nio1][T0][Y0-0000000000000000-0-0] [lt=5] connection close(sessid=3221553317, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.142213] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=11][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.142209] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] new mysql sessid created(conn.sessid_=3221553319, support_ssl=false) [2025-05-15 09:40:43.142223] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=10][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.142238] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=8] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142251] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=11][errcode=-4018] get session fail(ret=-4018, sessid=3221553318, proxy_sessid=0) [2025-05-15 09:40:43.142253] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] sm conn init succ(conn.sessid_=3221553319, sess.client_addr_="172.22.0.1:40612") [2025-05-15 09:40:43.142272] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=9][errcode=0] server close connection(sessid=3221553318, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142281] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=9][errcode=-4018] get session fail(ret=-4018, sessid=3221553318, proxy_sessid=0) [2025-05-15 09:40:43.142290] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=9][errcode=-4016] session is null [2025-05-15 09:40:43.142276] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=20] accept one succ(*s={this:0x7f1a6f616030, session_id:3221553319, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:130}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.142298] INFO [SERVER] process (obmp_connect.cpp:504) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B8-0-0] [lt=7] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553318, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.142312] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553320, support_ssl=false) [2025-05-15 09:40:43.142358] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] sm conn init succ(conn.sessid_=3221553320, sess.client_addr_="172.22.0.1:40618") [2025-05-15 09:40:43.142378] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=18] accept one succ(*s={this:0x7f1a6f604030, session_id:3221553320, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:146}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.142427] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f616db0, session_id:3221553318, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.142459] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=28] kill and revert session(conn.sessid_=3221553318, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.142476] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] can close safely, do destroy(*s={this:0x7f1a6f616db0, session_id:3221553318, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:145}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:230, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.142523] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=45] mark session id unused(conn.sessid_=3221553318) [2025-05-15 09:40:43.142555] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] connection close(sessid=3221553318, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.142604] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=33] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.142629] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=12] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.142647] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.142647] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.142665] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4130][sql_nio2][T1][Y0-0000000000000000-0-0] [lt=11] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.142665] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.142674] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=8][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.142682] WDIAG [SERVER] process (obmp_connect.cpp:333) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=7][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.142678] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=21][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.142698] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=19][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.142706] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=15] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142715] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=16][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.142721] WDIAG [SERVER] process (obmp_connect.cpp:333) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=6][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.142722] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553319, proxy_sessid=0) [2025-05-15 09:40:43.142738] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142752] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=12][errcode=-4018] get session fail(ret=-4018, sessid=3221553320, proxy_sessid=0) [2025-05-15 09:40:43.142755] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=15][errcode=0] server close connection(sessid=3221553319, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142770] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=8][errcode=0] server close connection(sessid=3221553320, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.142773] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=17][errcode=-4018] get session fail(ret=-4018, sessid=3221553319, proxy_sessid=0) [2025-05-15 09:40:43.142779] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=8][errcode=-4018] get session fail(ret=-4018, sessid=3221553320, proxy_sessid=0) [2025-05-15 09:40:43.142791] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=12][errcode=-4016] session is null [2025-05-15 09:40:43.142792] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:866) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=18][errcode=-4016] session is null [2025-05-15 09:40:43.142799] INFO [SERVER] process (obmp_connect.cpp:504) [4324][T1_MysqlQueueTh][T1][Y0-000635291BFC90AA-0-0] [lt=6] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553320, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.142802] INFO [SERVER] process (obmp_connect.cpp:504) [4322][T1_MysqlQueueTh][T1][Y0-000635291BDC90E8-0-0] [lt=8] MySQL LOGIN(direct_client_ip="172.22.0.1", client_ip=, tenant_name=, tenant_id=1, user_name=root, host_name=xxx.xxx.xxx.xxx, sessid=3221553319, proxy_sessid=0, client_sessid=4294967295, from_proxy=false, from_java_client=false, from_oci_client=false, from_jdbc_client=false, capability=270377487, proxy_capability=0, use_ssl=false, c/s protocol="OB_MYSQL_CS_TYPE", autocommit=false, proc_ret=-8001, ret=0, conn->client_type_=3, conn->client_version_=0) [2025-05-15 09:40:43.142838] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f604030, session_id:3221553320, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.142865] WDIAG [RPC.OBMYSQL] push_close_req (ob_sql_nio.cpp:881) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=21][errcode=-4015] close sql sock by user req(*s={this:0x7f1a6f616030, session_id:3221553319, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.142884] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=16] kill and revert session(conn.sessid_=3221553320, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.142895] INFO [RPC.OBMYSQL] on_disconnect (obsm_conn_callback.cpp:267) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=11] kill and revert session(conn.sessid_=3221553319, proxy_sessid=0, ret=0) [2025-05-15 09:40:43.142903] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=8] can close safely, do destroy(*s={this:0x7f1a6f604030, session_id:3221553320, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:146}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.142920] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=17] mark session id unused(conn.sessid_=3221553320) [2025-05-15 09:40:43.142931] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553320, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.142951] INFO [RPC.OBMYSQL] handle_pending_destroy_list (ob_sql_nio.cpp:982) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=19] can close safely, do destroy(*s={this:0x7f1a6f616030, session_id:3221553319, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:256, sql_initiative_shutdown:true, reader:{fd:130}, err:5, last_decode_time:1747302043141893, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:234, pending_flag:1, may_handling_flag:false, handler_close_flag:false}) [2025-05-15 09:40:43.142967] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:219) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=15] mark session id unused(conn.sessid_=3221553319) [2025-05-15 09:40:43.142976] INFO [RPC.OBMYSQL] destroy (obsm_conn_callback.cpp:243) [4130][sql_nio2][T0][Y0-0000000000000000-0-0] [lt=6] connection close(sessid=3221553319, proxy_sessid=0, tenant_id=1, from_proxy=false, from_java_client=false, c/s protocol="OB_MYSQL_CS_TYPE", is_need_clear_sessid_=true, is_sess_alloc_=false, ret=0, trace_id=Y0-0000000000000000-0-0, conn.pkt_rec_wrapper_=[start_pkt_pos_:0, cur_pkt_pos_:1, pkt_rec[0]:{send:obp_mysql_header_{len_:102, seq_:3}, pkt_name:"PKT_ERR", obp_mysql_header_.is_send_:1}], disconnect_state=0) [2025-05-15 09:40:43.143556] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4292][T1_TimerWK3_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=0] sql audit evict task end(request_manager_->get_tenant_id()=1, evict_high_mem_level=128849018, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=72791040) [2025-05-15 09:40:43.143734] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=21] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:43.146927] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.148941] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.148966] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.148983] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.148998] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.149011] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.149026] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.149040] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.149054] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.149069] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.149081] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.149096] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.149114] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043149112, replica_locations:[]}) [2025-05-15 09:40:43.149131] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:43.149156] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:43.149169] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:43.149196] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.149247] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11807039722, cache_obj->added_lc()=false, cache_obj->get_object_id()=15855, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.150393] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.150414] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.150428] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.150441] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.150449] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.150462] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.150477] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.150502] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.150515] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.150527] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.150541] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.150554] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043150553, replica_locations:[]}) [2025-05-15 09:40:43.150615] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=1121066, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=333, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:43.157040] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.158844] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:43.167155] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=20][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.173937] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4111][RpcIO][T0][Y0-0000000000000000-0-0] [lt=14] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.174089] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4113][RpcIO][T0][Y0-0000000000000000-0-0] [lt=17] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.174293] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4112][RpcIO][T0][Y0-0000000000000000-0-0] [lt=20] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.175043] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4114][RpcIO][T0][Y0-0000000000000000-0-0] [lt=21] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.177264] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.187371] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.192298] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4339][T1_HAService][T1][Y0-0000000000000000-0-0] [lt=16] start do ls ha handler(ls_id_array_=[{id:1}]) [2025-05-15 09:40:43.194324] INFO [STORAGE] runTimerTask (ob_checkpoint_service.cpp:289) [4289][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=0] ====== check clog disk timer task ====== [2025-05-15 09:40:43.194343] INFO [PALF] get_disk_usage (palf_env_impl.cpp:902) [4289][T1_TimerWK0_CKClogDisk][T1][Y0-0000000000000000-0-0] [lt=15] get_disk_usage(ret=0, capacity(MB):=5120, used(MB):=2872) [2025-05-15 09:40:43.197479] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.205426] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:43.205464] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=35][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:43.205480] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=14][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:43.205502] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=21][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:43.205516] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=10][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:43.205523] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=7][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:43.205535] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=6][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:43.205545] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:43.205557] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=11][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:43.205564] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=6][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:43.205573] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=8][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:43.205584] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:43.205594] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=8][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:43.205603] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=8][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:43.205626] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:43.205636] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:43.205648] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:43.205658] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:43.205669] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:43.205682] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=11][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:43.205692] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=8][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:43.205712] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:43.205735] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=17][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:43.205744] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:43.205753] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:43.205770] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=8][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:43.205784] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4462][T1001_TimerWK2_LSMetaCh][T1][YB427F000001-000635291DBC6456-0-0] [lt=11][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.205796] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=11][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:43.205808] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=10][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.205818] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=10][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:43.205828] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:43.205838] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78cada0, start=1747302043205157, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.205850] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=11][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:43.205861] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1001 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.205967] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=10][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:43.205993] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=26][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1001, inner_table_only=false) [2025-05-15 09:40:43.206003] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:43.206013] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:43.206029] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=12][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:43.206042] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=10][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=0) [2025-05-15 09:40:43.206051] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4462][T1001_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291DBC6456-0-0] [lt=9][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:43.207600] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=27][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.207913] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:43.208738] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.208779] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=27][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043208731) [2025-05-15 09:40:43.208794] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302043008673, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.208831] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12345457358, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.208851] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.208875] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.208892] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.208903] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.208918] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12345457358, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.208931] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.208944] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.208955] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:43.208964] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:43.208976] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.208986] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.208995] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043208807) [2025-05-15 09:40:43.217704] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.220688] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.220711] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=22][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:43.220732] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:43.220747] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.220760] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:43.220772] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.220782] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.220792] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:43.220803] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302043220670, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.220815] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:43.220826] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.220875] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:43.220886] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:43.220898] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:43.220908] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:43.227812] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.230442] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.230469] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=26][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043230436) [2025-05-15 09:40:43.230483] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302043030387, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.230516] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20] get wrs ts(ls_id={id:1}, delta=12345501834, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:43.230517] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15] get wrs ts(ls_id={id:1}, delta=12345329341, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:43.230534] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.230534] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=14] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.230553] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:43.230554] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=18][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:43.230567] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.230569] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.230575] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.230577] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.230588] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.230595] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.230601] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043230503) [2025-05-15 09:40:43.230675] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=97][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.230681] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.230689] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043230503) [2025-05-15 09:40:43.237926] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.248034] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.250754] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.250777] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.250795] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.250809] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.250822] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.250836] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.250852] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.250865] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.250878] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.250889] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.250904] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.250919] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043250918, replica_locations:[]}) [2025-05-15 09:40:43.250937] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:43.250960] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:43.250972] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:43.251010] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.251060] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11807141536, cache_obj->added_lc()=false, cache_obj->get_object_id()=15856, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.251319] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=20] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=2) [2025-05-15 09:40:43.252316] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.252337] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.252351] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.252364] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.252377] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.252393] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.252407] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.252420] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.252433] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.252444] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.252458] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.252471] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043252470, replica_locations:[]}) [2025-05-15 09:40:43.252544] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=1019138, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=334, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:43.257173] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.257193] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=19][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:43.257202] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1001, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:43.257218] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1001, mem_limit=431930736, config_mem_limit_=16777216) [2025-05-15 09:40:43.257233] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4463][T1001_TimerWK3_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=12] sql audit evict task end(request_manager_->get_tenant_id()=1001, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=4159488) [2025-05-15 09:40:43.257262] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4461][T1001_TimerWK1_ReqMemEvict][T1001][Y0-0000000000000000-0-0] [lt=8] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:43.258143] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.259841] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:178) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=32] tmp file swap statistics(swap_task_cnt=0, avg_swap_response_time=0, max_swap_response_time=-1, min_swap_response_time=-1) [2025-05-15 09:40:43.259868] INFO [STORAGE] print_block_usage (ob_tmp_file_block_manager.cpp:822) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=21] temporary file module use no blocks [2025-05-15 09:40:43.259879] INFO [STORAGE] print_statistics (ob_tmp_file_thread_job.cpp:124) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=9] tmp file flush statistics(flush_task_cnt=0, avg_flush_data_len=0, max_flush_data_len=-1, min_flush_data_len=-1, f1_cnt=0, f2_cnt=0, f3_cnt=0, f4_cnt=0, f5_cnt=0) [2025-05-15 09:40:43.259896] INFO [STORAGE] print_statistics (ob_tmp_file_write_buffer_pool.cpp:1595) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=13] tmp file write buffer pool statistics(dirty_page_percentage=0, max_page_num=6578, fat_actual_size=0, dirty_page_num=0, total_write_back_num=0, meta_page_num=0, dirty_meta_page_num=0, write_back_meta_num=0, data_page_num=0, dirty_data_page_num=0, write_back_data_num=0, data_page_watermark=0, meta_page_watermark=0) [2025-05-15 09:40:43.259913] INFO [STORAGE] do_work_ (ob_tmp_file_thread_wrapper.cpp:305) [4317][T1_TFSwap][T1][Y0-0000000000000000-0-0] [lt=17] ObTmpFileFlushTG information(this={is_inited_:true, mode_:1, last_flush_timestamp_:11806150262, flush_io_finished_ret_:0, flush_io_finished_round_:1517, flushing_block_num_:0, is_fast_flush_meta_:false, fast_flush_meta_task_cnt_:0, wait_list_size_:0, retry_list_size_:0, finished_list_size_:0, normal_loop_cnt_:2, normal_idle_loop_cnt_:2, fast_loop_cnt_:0, fast_idle_loop_cnt_:0, flush_mgr_:{is_inited_:true, flush_ctx_:{is_inited_:true, fail_too_many_:false, expect_flush_size_:0, actual_flush_size_:0, flush_seq_ctx_:{create_flush_task_cnt_:0, prepare_finished_cnt_:0, flush_sequence_:0}, state_:5, iter_:{is_inited_:false, cur_caching_list_idx_:0, cur_caching_list_is_meta_:false, cur_iter_dir_idx_:-1, cur_iter_file_idx_:-1, cached_file_num_:0, cached_dir_num_:0}, file_ctx_hash_.size():0, flush_failed_array_.size():0}}}) [2025-05-15 09:40:43.264954] INFO [STORAGE] gc_tables_in_queue (ob_tenant_meta_mem_mgr.cpp:610) [4292][T1_TimerWK3_TenantMetaMemMg][T1][Y0-0000000000000000-0-0] [lt=18] gc tables in queue: recycle 0 table(ret=0, tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl3784EEE", sizeof(T):3816, used_obj_cnt:1300, free_obj_hold_cnt:1, allocator used:4995840, allocator total:5352960}, large_tablet_buffer_pool_={typeid(T).name():"N9oceanbase7storage15ObMetaObjBufferINS0_8ObTabletELl65448EEE", sizeof(T):65480, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, full_tablet_creator_={tiny_allocator_.used():0, tiny_allocator_.total():0, full allocator used:0, full allocator total:0}, tablets_mem=5352960, tablets_mem_limit=1073741820, ddl_kv_pool_={typeid(T).name():"N9oceanbase7storage7ObDDLKVE", sizeof(T):1024, used_obj_cnt:0, free_obj_hold_cnt:0, allocator used:0, allocator total:0}, memtable_pool_={typeid(T).name():"N9oceanbase8memtable10ObMemtableE", sizeof(T):1984, used_obj_cnt:87, free_obj_hold_cnt:0, allocator used:178176, allocator total:195840}, pending_cnt=0, wait_gc_count=0, tablet count=1300) [2025-05-15 09:40:43.268249] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.271754] INFO [COORDINATOR] detect_recover (ob_failure_detector.cpp:127) [4420][T1001_Occam][T1001][Y0-0000000000000000-0-0] [lt=26] doing detect recover operation(events_with_ops=[{event:{type:SCHEMA NOT REFRESHED, module:SCHEMA, info:schema not refreshed, level:SERIOUS}}]) [2025-05-15 09:40:43.278357] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.288466] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.298928] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.307971] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:43.308777] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12345556697, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.308809] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.308840] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=28][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.308858] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.308870] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.308887] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12345556697, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.308892] WDIAG load_file_to_string (utility.h:673) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=20][errcode=0] read /sys/class/net/lo/speed failed, errno 22 [2025-05-15 09:40:43.308898] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.308908] WDIAG get_ethernet_speed (utility.cpp:575) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=14][errcode=-4000] load file /sys/class/net/lo/speed failed, ret -4000 [2025-05-15 09:40:43.308917] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=19][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.308919] WDIAG [SERVER] get_network_speed_from_sysfs (ob_server.cpp:3179) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4000] cannot get Ethernet speed, use default(tmp_ret=0, devname="lo") [2025-05-15 09:40:43.308930] WDIAG [SERVER] runTimerTask (ob_server.cpp:3722) [3956][TimerWK3_ServerGTimer][T0][Y0-0000000000000000-0-0] [lt=10][errcode=-4000] ObRefreshNetworkSpeedTask reload bandwidth throttle limit failed(ret=-4000, ret="OB_ERROR") [2025-05-15 09:40:43.308930] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:43.308937] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:43.308957] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.308977] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.308995] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043308753) [2025-05-15 09:40:43.309564] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.319678] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.320822] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.320841] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:43.320861] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:43.320877] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.320891] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:43.320904] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.320918] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.320929] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:43.320940] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302043320797, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.320953] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:43.320964] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.321032] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:43.321046] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:43.321054] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:43.321065] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:43.325560] WDIAG [RPC] check_blacklist (ob_poc_rpc_proxy.cpp:273) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=41][errcode=-4122] address in blacklist(ret=-4122, addr="172.19.0.2:2882") [2025-05-15 09:40:43.325578] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.325593] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=9][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.325612] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.325629] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=17][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.325643] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.325659] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=15][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.325679] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=16][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.325698] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.325714] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=15][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.325727] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=12] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.325745] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=13] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.325777] INFO [SERVER] check_report_replicas_ (ob_tenant_meta_checker.cpp:587) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=13] add missing replica to ls meta table success(ret=0, ret="OB_SUCCESS", tenant_id=1, ls_id={id:1}) [2025-05-15 09:40:43.325791] INFO [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:226) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=12] checker found and corrected dangling or to report replicas for ls meta table(ret=0, ret="OB_SUCCESS", tenant_id=1, dangling_count=0, report_count=1, ls_checker_tg_id=310) [2025-05-15 09:40:43.327293] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=12] table not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, table_name.ptr()="data_size:19, data:5F5F616C6C5F6C735F6D6574615F7461626C65", ret=-5019) [2025-05-15 09:40:43.327319] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=22][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_ls_meta_table, ret=-5019) [2025-05-15 09:40:43.327331] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_ls_meta_table, db_name=oceanbase) [2025-05-15 09:40:43.327342] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=10][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_ls_meta_table) [2025-05-15 09:40:43.327357] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=12][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:43.327368] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:43.327383] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=10][errcode=-5019] Table 'oceanbase.__all_ls_meta_table' doesn't exist [2025-05-15 09:40:43.327415] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=31][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:43.327428] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=13][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:43.327436] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=6][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:43.327447] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=10][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:43.327462] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=14][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:43.327470] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=7][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:43.327481] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:43.327510] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=20][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:43.327522] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:43.327532] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=7][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:43.327543] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=9][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:43.327553] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=8][errcode=-5019] fail to handle text query(stmt=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port, ret=-5019) [2025-05-15 09:40:43.327569] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=13][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:43.327576] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=6][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:43.327594] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=13][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:43.327615] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=18][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:43.327627] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:43.327633] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=6][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:43.327649] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:43.327661] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.327680] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=18][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5019, ret=-5019) [2025-05-15 09:40:43.327694] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=13][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.327701] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=7][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:43.327711] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:43.327722] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=10][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca700, start=1747302043327061, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.327731] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=9][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:43.327738] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=6][errcode=-5019] execute sql failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.327809] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=-5019] get all ls info by persistent_ls_ failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1) [2025-05-15 09:40:43.327829] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=20][errcode=-5019] fail to get ls infos by tenant(ret=-5019, ret="OB_TABLE_NOT_EXIST", tenant_id=1, inner_table_only=true) [2025-05-15 09:40:43.327841] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=-5019] fail to open iterator(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:43.327853] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=12][errcode=-5019] ls table iterator next failed(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:43.327862] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=6][errcode=-5019] build replica map from ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:43.327873] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:199) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=10][errcode=-5019] check ls table failed(ret=-5019, ret="OB_TABLE_NOT_EXIST", mode=1) [2025-05-15 09:40:43.327885] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4292][T1_TimerWK3_LSMetaCh][T1][YB427F000001-000635291C5C6393-0-0] [lt=11][errcode=-5019] fail to check ls meta table(ret=-5019, ret="OB_TABLE_NOT_EXIST") [2025-05-15 09:40:43.329789] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.330516] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=7] get wrs ts(ls_id={id:1}, delta=12345602254, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:43.330540] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.330564] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:43.330568] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.330580] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.330591] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.330604] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.330606] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=22][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043330562) [2025-05-15 09:40:43.330617] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.330619] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302043130534, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.330624] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043330499) [2025-05-15 09:40:43.330650] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12345429761, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:43.330666] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.330684] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:43.330698] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.330709] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.330723] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.330730] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=7][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.330740] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043330639) [2025-05-15 09:40:43.339895] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.350004] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.352675] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.352699] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=21][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.352716] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.352729] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.352741] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.352755] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.352774] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.352793] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.352807] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.352818] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.352829] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.352845] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043352843, replica_locations:[]}) [2025-05-15 09:40:43.352864] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:43.352888] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:43.352899] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:43.352926] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=1][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.352975] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11807243451, cache_obj->added_lc()=false, cache_obj->get_object_id()=15857, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.354155] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.354176] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.354191] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.354203] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.354214] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.354226] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.354240] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.354250] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=7][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.354262] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.354273] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.354286] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.354298] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043354298, replica_locations:[]}) [2025-05-15 09:40:43.354359] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=917322, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=335, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:43.356423] INFO [STORAGE] scheduler_ls_ha_handler_ (ob_storage_ha_service.cpp:195) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=10] start do ls ha handler(ls_id_array_=[{id:1001}, {id:1}]) [2025-05-15 09:40:43.356456] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.356468] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:43.356480] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:43.356504] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=23][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1001}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747289697751194001, v:0}, clog_base_lsn:{lsn:738152448}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1001}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1001, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:43.356586] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=82][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1001}) [2025-05-15 09:40:43.356596] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.356605] WDIAG [STORAGE] check_meta_tenant_normal_ (ob_ls_restore_handler.cpp:407) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] failed to get schema guard(ret=-5627, meta_tenant_id=1001) [2025-05-15 09:40:43.356611] WDIAG [STORAGE] check_before_do_restore_ (ob_ls_restore_handler.cpp:319) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-5627] fail to get meta tenant status(ret=-5627) [2025-05-15 09:40:43.356620] WDIAG [STORAGE] process (ob_ls_restore_handler.cpp:277) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5627] fail to check before do restore(ret=-5627, ls_={running_state:{state:1}, ls_meta:{tenant_id:1002, ls_id:{id:1}, ls_persistent_state:{state:1}, clog_checkpoint_scn:{val:1747274406063927000, v:0}, clog_base_lsn:{lsn:872361984}, rebuild_seq:0, migration_status:0, gc_state_:1, offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:0}, replayable_point:{val:1747289695220131001, v:0}, tablet_change_checkpoint_scn:{val:1747289697751194001, v:0}, all_id_meta:{id_meta:[{limited_id:1747289705606264000, latest_log_ts:{val:1747289689210721001, v:0}}, {limited_id:6000001, latest_log_ts:{val:1747186185606264001, v:0}}, {limited_id:1, latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:0, v:0}, rebuild_info:{status:{status:0}, type:{type:0}, tablet_id_array:{id_array:[]}, src:[type = 0], [address = ]}, transfer_meta_info:{src_ls:{id:0}, src_scn:{val:1, v:0}, trans_status:0, tablet_id_array:{id_array:[]}, data_version:1(0, 0, 1)}, store_format:{store_type:1, store_type_str:"NORMAL"}}, switch_epoch:1, log_handler:{role:2, proposal_id:4, palf_env_:0x7f1a8fdf6030, is_in_stop_state_:false, is_inited_:true, id_:1}, restore_handler:{is_inited:true, is_in_stop_state:false, id:1, proposal_id:9223372036854775807, role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1, max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615}, max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0, trace_id:Y0-0000000000000000-0-0, error_type:0, err_lsn:{lsn:18446744073709551615}}, task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true, tablet_gc_handler:{tablet_persist_trigger:0, is_inited:true}, startup_transfer_info:{ls_id:{id:0}, transfer_start_scn:{val:18446744073709551615, v:3}}, need_delay_resource_recycle:false}) [2025-05-15 09:40:43.356674] WDIAG [STORAGE] do_ha_handler_ (ob_storage_ha_service.cpp:232) [4637][T1002_HAService][T1002][Y0-0000000000000000-0-0] [lt=55][errcode=0] failed to do ls restore handler process(tmp_ret=-5627, ls_id={id:1}) [2025-05-15 09:40:43.359130] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=27] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:43.360119] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=22][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.364442] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=17] run rewrite rule refresh task(rule_mgr_->tenant_id_=1002) [2025-05-15 09:40:43.365859] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=16][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.365879] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=20][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.365890] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=10][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.365903] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=13][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.365914] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=10][errcode=-5157] get schema guard failed(ret=-5157) [2025-05-15 09:40:43.365938] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=11][errcode=-5157] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5157, ret=-5157) [2025-05-15 09:40:43.365953] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=12][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.365966] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_ReqMemEvict][T1002][YB427F000001-000635291F9C666F-0-0] [lt=12][errcode=-5157] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5157, ret=-5157) [2025-05-15 09:40:43.365978] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] execute sql failed(ret=-5157, tenant_id=1002, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:43.365990] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] retry_while_no_tenant_resource failed(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.365998] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] execute_read failed(ret=-5157, cluster_id=1747291386, tenant_id=1002) [2025-05-15 09:40:43.366009] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] query failed(ret=-5157, conn=0x7f1ae78cb440, start=1747302043365837, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:43.366023] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-5157] read failed(ret=-5157) [2025-05-15 09:40:43.366036] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5157) [2025-05-15 09:40:43.366101] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-5157] failed to fetch max rule version(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.366113] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-5157] failed to fetch max rule version(ret=-5157) [2025-05-15 09:40:43.366120] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4579][T1002_TimerWK2_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] failed to sync rule from inner table(ret=-5157) [2025-05-15 09:40:43.370228] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.370521] INFO [COMMON] print_sender_status (ob_io_manager.cpp:1413) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=18] [IO STATUS SENDER](io_status=req_in_sender, total=0, total_local_read=0, total_local_write=0, total_remote_read=0, total_remote_write=0, cnt=0 0 0 0 0 0 0 0 , local_read=0 0 0 0 0 0 0 0 , local_write=0 0 0 0 0 0 0 0 , remote_read=0 0 0 0 0 0 0 0 , remote_write=0 0 0 0 0 0 0 0 ) [2025-05-15 09:40:43.370588] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=16] [IO STATUS CALLBACK](tenant_id=1, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:342, idx_:0, tid_:4295}], io_callback_status=runner 0: thread_id=4295, queue_count=0, ) [2025-05-15 09:40:43.370653] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=19] [IO STATUS CALLBACK](tenant_id=1001, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:402, idx_:0, tid_:4466}], io_callback_status=runner 0: thread_id=4466, queue_count=0, ) [2025-05-15 09:40:43.370714] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=14] [IO STATUS CALLBACK](tenant_id=1002, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:462, idx_:0, tid_:4583}], io_callback_status=runner 0: thread_id=4583, queue_count=0, ) [2025-05-15 09:40:43.370746] INFO [COMMON] print_io_status (ob_io_manager.cpp:2637) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS GROUP](tenant_id=500, io_status=group_id:0, group_name:OTHER_GROUPS, mode:LOCAL READ, cur_req:0, hold_mem:0 [FAILED]:fail_size:0, fail_iops:0, fail_bw:0, [delay/us]:prepare:0, schedule:0, submit:0, rt:0, total:0, [SUCC]:size:4096, iops:1, norm_iops:0, bw:4091, limit:50000, [delay/us]:prepare:0, schedule:0, submit:600000000, rt:14491, total:14501) [2025-05-15 09:40:43.370782] INFO [COMMON] print_io_status (ob_io_manager.cpp:2761) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS TENANT](tenant_id=500, ref_cnt=1, io_config={group_configs_cnt:3, memory_limit_:536870912, callback_thread_count_:0, unit_config_:{min_iops:10000, max_iops:50000, weight:10000, max_net_bandwidth:9223372036854775807, net_bandwidth_weight:100}, enable_io_tracer:false, object_storage_io_timeout_ms:20000, group_configs:[group_id = 0:{mode:0, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:1, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}, group_id = 0:{mode:2, deleted:false, cleared:false, min_percent:100, max_percent:100, weight_percent:100}]}, hold_mem=31368256, free_req_cnt=50000, free_result_cnt=50000, callback_queues=[0], [FAILED]: fail_ips=0, fail_ops=0, fail_iops=0, fail_ibw=0, fail_obw=0, [SUCC]: ips=0, ops=0, iops=0, ibw=0, obw=0, iops_limit=50000, ibw_limit=9223372036854775807, obw_limit=9223372036854775807) [2025-05-15 09:40:43.370838] INFO [COMMON] print_io_function_status (ob_io_manager.cpp:2841) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=52] [IO STATUS FUNCTION](tenant_id=500, io_status=function_name:OTHER_GROUPS, mode:LOCAL READ, avg_size:4096, avg_iops:1, avg_bw:4091, [delay/us]: prepare:0, schedule:0, submit:600000000, device:14491, total:14501) [2025-05-15 09:40:43.370869] INFO [COMMON] print_io_status (ob_io_manager.cpp:2783) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CALLBACK](tenant_id=500, runners=[{is_inited_:true, queue_.get_total():0, tg_id_:285, idx_:0, tid_:4132}], io_callback_status=runner 0: thread_id=4132, queue_count=0, ) [2025-05-15 09:40:43.370887] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=11] [IO STATUS CHANNEL] device_id=0x7f1ae77ad550, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:43.370903] INFO [COMMON] print_status (ob_io_struct.cpp:2860) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=12] [IO STATUS CHANNEL] device_id=0x7f1ae77ac750, req_in_channel, async_total=0, sync_total=0, async_cnt= 0 0 0 0 0 0 0 0, sync_cnt= 0 [2025-05-15 09:40:43.370924] INFO [STORAGE.BLKMGR] alloc_block (ob_block_manager.cpp:336) [4133][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=6] successfully alloc block(macro_id={[ver=1,mode=0,seq=628778][2nd=1905]}) [2025-05-15 09:40:43.380135] INFO [LIB] log_compress_loop_ (ob_log_compressor.cpp:410) [3960][SyslogCompress][T0][Y0-0000000000000000-0-0] [lt=25] log compressor cycles once. (ret=0, cost_time=0, compressed_file_count=0, deleted_file_count=0, disk_remaining_size=3626806108160, fast_delete_log_mode=false) [2025-05-15 09:40:43.380309] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.390426] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.395857] INFO [RPC.FRAME] mysql_easy_timer_cb (ob_net_easy.cpp:674) [4122][MysqlUnix][T0][Y0-0000000000000000-0-0] [lt=18] [MYSQL EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.395980] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4116][BatchIO][T0][Y0-0000000000000000-0-0] [lt=21] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.396126] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4117][BatchIO][T0][Y0-0000000000000000-0-0] [lt=13] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.396389] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4120][BatchIO][T0][Y0-0000000000000000-0-0] [lt=43] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.396411] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4119][BatchIO][T0][Y0-0000000000000000-0-0] [lt=20] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.396502] INFO [RPC.FRAME] batch_rpc_easy_timer_cb (ob_net_easy.cpp:639) [4118][BatchIO][T0][Y0-0000000000000000-0-0] [lt=17] [BATCH_RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.396529] INFO [RPC.FRAME] rpc_easy_timer_cb (ob_net_easy.cpp:601) [4124][RpcUnix][T0][Y0-0000000000000000-0-0] [lt=30] [RPC EASY STAT](log_str=conn count=0/0, request done=0/0, request doing=0/0) [2025-05-15 09:40:43.398516] INFO [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:86) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=17] run rewrite rule refresh task(rule_mgr_->tenant_id_=1) [2025-05-15 09:40:43.398550] INFO [SQL.PC] update_memory_conf (ob_plan_cache.cpp:1963) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=10] update plan cache memory config(ob_plan_cache_percentage=5, ob_plan_cache_evict_high_percentage=90, ob_plan_cache_evict_low_percentage=50, tenant_id=1) [2025-05-15 09:40:43.398610] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1465) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=54] start lib cache evict(tenant_id=1, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:43.398630] INFO [SQL.PC] cache_evict (ob_plan_cache.cpp:1482) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=18] end lib cache evict(tenant_id=1, cache_evict_num=0, mem_hold=2097152, mem_limit=268435455, cache_obj_num=0, cache_node_num=0) [2025-05-15 09:40:43.398646] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2750) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=15] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:43.400551] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=18][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.400606] INFO [SQL.RESV] check_table_exist_or_not (ob_dml_resolver.cpp:11287) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=22] table not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, table_name.ptr()="data_size:14, data:5F5F616C6C5F7379735F73746174", ret=-5019) [2025-05-15 09:40:43.400639] WDIAG [SQL.RESV] resolve_table_relation_recursively (ob_dml_resolver.cpp:11235) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=30][errcode=-5019] synonym not exist(tenant_id=1, database_id=201001, table_name=__all_sys_stat, ret=-5019) [2025-05-15 09:40:43.400658] WDIAG [SQL.RESV] resolve_table_relation_factor_normal (ob_dml_resolver.cpp:11065) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=17][errcode=-5019] fail to resolve table relation recursively(tenant_id=1, ret=-5019, database_id=201001, database_id=201001, table_name=__all_sys_stat, db_name=oceanbase) [2025-05-15 09:40:43.400676] WDIAG [SQL.RESV] resolve_table_relation_factor (ob_dml_resolver.cpp:10903) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=16][errcode=-5019] resolve table relation factor failed(ret=-5019, table_name=__all_sys_stat) [2025-05-15 09:40:43.400693] WDIAG [SQL.RESV] inner_resolve_sys_view (ob_dml_resolver.cpp:3414) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=13][errcode=-5019] fail to resolve table(ret=-5019) [2025-05-15 09:40:43.400707] WDIAG [SQL.RESV] resolve_table_relation_factor_wrapper (ob_dml_resolver.cpp:3469) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=13][errcode=-5019] fail to resolve sys view(ret=-5019) [2025-05-15 09:40:43.400724] WDIAG resolve_basic_table_without_cte (ob_dml_resolver.cpp:3565) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=12][errcode=-5019] Table 'oceanbase.__all_sys_stat' doesn't exist [2025-05-15 09:40:43.400715] INFO [SQL.PC] dump_all_objs (ob_plan_cache.cpp:2468) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=12] Dumping All Cache Objs(alloc_obj_list.count()=1, alloc_obj_list=[{obj_id:15858, tenant_id:1, log_del_time:9223372036854775807, real_del_time:9223372036854775807, ref_count:1, added_to_lc:false, mem_used:25832}]) [2025-05-15 09:40:43.400737] WDIAG [SQL.RESV] resolve_basic_table_with_cte (ob_dml_resolver.cpp:18648) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=11][errcode=-5019] resolve base or alias table factor failed(ret=-5019) [2025-05-15 09:40:43.400738] INFO [SQL.PC] runTimerTask (ob_plan_cache.cpp:2758) [4292][T1_TimerWK3_PlanCacheEvict][T1][Y0-0000000000000000-0-0] [lt=22] schedule next cache evict task(evict_interval=5000000) [2025-05-15 09:40:43.400752] WDIAG [SQL.RESV] resolve_basic_table (ob_dml_resolver.cpp:18536) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=14][errcode=-5019] fail to resolve basic table with cte(ret=-5019) [2025-05-15 09:40:43.400767] WDIAG [SQL.RESV] resolve_table (ob_dml_resolver.cpp:5405) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=15][errcode=-5019] failed to resolve basic table(ret=-5019) [2025-05-15 09:40:43.400779] WDIAG [SQL.RESV] resolve_from_clause (ob_select_resolver.cpp:3944) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=12][errcode=-5019] fail to exec resolve_table(*table_node, table_item)(ret=-5019) [2025-05-15 09:40:43.400789] WDIAG [SQL.RESV] resolve_normal_query (ob_select_resolver.cpp:1276) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=9][errcode=-5019] fail to exec resolve_from_clause(parse_tree.children_[PARSE_SELECT_FROM])(ret=-5019) [2025-05-15 09:40:43.400800] WDIAG [SQL.RESV] resolve (ob_select_resolver.cpp:1502) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] resolve normal query failed(ret=-5019) [2025-05-15 09:40:43.400810] WDIAG [SQL.RESV] select_stmt_resolver_func (ob_resolver.cpp:186) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] execute stmt_resolver failed(ret=-5019, parse_tree.type_=3299) [2025-05-15 09:40:43.400833] WDIAG [SQL] generate_stmt (ob_sql.cpp:3065) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=11][errcode=-5019] failed to resolve(ret=-5019) [2025-05-15 09:40:43.400844] WDIAG [SQL] generate_physical_plan (ob_sql.cpp:3186) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] Failed to generate stmt(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:43.400857] WDIAG [SQL] handle_physical_plan (ob_sql.cpp:5118) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] Failed to generate plan(ret=-5019, result.get_exec_context().need_disconnect()=false) [2025-05-15 09:40:43.400869] WDIAG [SQL] handle_text_query (ob_sql.cpp:2774) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] fail to handle physical plan(ret=-5019) [2025-05-15 09:40:43.400881] WDIAG [SQL] stmt_query (ob_sql.cpp:174) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] fail to handle text query(stmt=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:43.400896] WDIAG [SERVER] do_query (ob_inner_sql_connection.cpp:786) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=12][errcode=-5019] executor execute failed(ret=-5019) [2025-05-15 09:40:43.400913] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:945) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=16][errcode=-5019] execute failed(ret=-5019, tenant_id=1, executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, retry_cnt=0, local_sys_schema_version=1, local_tenant_schema_version=1) [2025-05-15 09:40:43.400932] WDIAG [SERVER] after_func (ob_query_retry_ctrl.cpp:1068) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=14][errcode=-5019] [RETRY] check if need retry(v={force_local_retry:true, stmt_retry_times:0, local_retry_times:0, err_:-5019, err_:"OB_TABLE_NOT_EXIST", retry_type:0, client_ret:-5019}, need_retry=false) [2025-05-15 09:40:43.400953] WDIAG [SERVER] inner_close (ob_inner_sql_result.cpp:214) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=16][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:43.400963] WDIAG [SERVER] force_close (ob_inner_sql_result.cpp:194) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] result set close failed(ret=-5019) [2025-05-15 09:40:43.400970] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:950) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=7][errcode=-5019] failed to close result(close_ret=-5019, ret=-5019) [2025-05-15 09:40:43.400993] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] failed to process record(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, record_ret=-5019, ret=-5019) [2025-05-15 09:40:43.401008] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.401018] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4291][T1_TimerWK2_ReqMemEvict][T1][YB427F000001-000635291D0C63C2-0-0] [lt=10][errcode=-5019] failed to process final(executor={ObIExecutor:, sql:"SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE"}, aret=-5019, ret=-5019) [2025-05-15 09:40:43.401031] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] execute sql failed(ret=-5019, tenant_id=1, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:43.401042] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] retry_while_no_tenant_resource failed(ret=-5019, tenant_id=1) [2025-05-15 09:40:43.401052] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-5019] execute_read failed(ret=-5019, cluster_id=1747291386, tenant_id=1) [2025-05-15 09:40:43.401063] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-5019] query failed(ret=-5019, conn=0x7f1ae78ca060, start=1747302043400268, sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE) [2025-05-15 09:40:43.401075] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] read failed(ret=-5019) [2025-05-15 09:40:43.401083] WDIAG [SHARE] fetch_max_id (ob_max_id_fetcher.cpp:505) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] execute sql failed(sql=SELECT VALUE FROM __all_sys_stat WHERE ZONE = '' AND NAME = 'ob_max_used_rewrite_rule_version' AND TENANT_ID = 0 FOR UPDATE, ret=-5019) [2025-05-15 09:40:43.401158] WDIAG [SQL.QRR] fetch_max_rule_version (ob_udr_sql_service.cpp:129) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=7][errcode=-5019] failed to fetch max rule version(ret=-5019, tenant_id=1) [2025-05-15 09:40:43.401171] WDIAG [SQL.QRR] sync_rule_from_inner_table (ob_udr_mgr.cpp:245) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-5019] failed to fetch max rule version(ret=-5019) [2025-05-15 09:40:43.401178] WDIAG [SQL.QRR] runTimerTask (ob_udr_mgr.cpp:88) [4291][T1_TimerWK2_ReqMemEvict][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-5019] failed to sync rule from inner table(ret=-5019) [2025-05-15 09:40:43.408026] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:43.408822] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1002", warnings:[]}, tenant_id_=1002, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.408854] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=29][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043408814) [2025-05-15 09:40:43.408871] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1002, last_post_cluster_heartbeat_tstamp_=1747302043208806, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.408919] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14] get wrs ts(ls_id={id:1}, delta=12345656960, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.408939] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.408964] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=22][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.408982] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.408993] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.409010] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1001}, delta=12345656960, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.409025] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.409039] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.409051] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:43.409059] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:43.409073] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.409084] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.409094] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043408895) [2025-05-15 09:40:43.410665] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.420769] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.420835] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=10][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.420855] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=20][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:43.420876] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:43.420892] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.420905] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:43.420918] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.420926] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.420940] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:43.420950] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cada0, start=1747302043420819, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.420963] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:43.420971] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=8][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.421022] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:43.421033] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=11][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:43.421047] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:43.421064] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:43.430565] WDIAG [STORAGE.TRANS] get_cluster_service_master_ (ob_tenant_weak_read_service.cpp:285) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4076] tenant schema is not ready, need wait(ret=-4076, ret="OB_NEED_WAIT", superior_tenant_id=1) [2025-05-15 09:40:43.430584] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=19][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1", warnings:[]}, tenant_id_=1, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.430603] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=18][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043430560) [2025-05-15 09:40:43.430619] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=16][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1, last_post_cluster_heartbeat_tstamp_=1747302043230502, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.430633] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1}, delta=12345529957, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:43.430642] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=13] get wrs ts(ls_id={id:1}, delta=12345702450, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:43.430653] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=17] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.430656] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.430674] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:43.430674] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=17][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:43.430689] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.430689] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.430697] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.430699] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.430712] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.430712] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.430721] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.430721] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.430728] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043430619) [2025-05-15 09:40:43.430727] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043430631) [2025-05-15 09:40:43.430878] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.440984] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.451097] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.451408] INFO [COMMON] replace_map (ob_kv_storecache.cpp:783) [3954][TimerWK1_KVCacheRep][T0][Y0-0000000000000000-0-0] [lt=18] replace map num details(ret=0, replace_node_count=0, map_once_replace_num_=100000, map_replace_skip_count_=1) [2025-05-15 09:40:43.454498] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=17][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.454521] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=19][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.454538] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.454551] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.454561] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.454575] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.454590] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.454604] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.454618] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.454629] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.454643] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.454658] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043454657, replica_locations:[]}) [2025-05-15 09:40:43.454675] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:43.454699] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:43.454709] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:43.454744] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.454791] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11807345268, cache_obj->added_lc()=false, cache_obj->get_object_id()=15858, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.455906] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.455928] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.455943] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.455956] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.455964] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=6][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.455976] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.455990] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.456003] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.456015] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.456025] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.456039] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.456052] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043456051, replica_locations:[]}) [2025-05-15 09:40:43.456112] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=815570, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=336, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:43.461211] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.471320] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.482393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.492505] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=19][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.501977] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4291][T1_TimerWK2_LogUpdater][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4721] REACH SYSLOG RATE LIMIT [bandwidth] [2025-05-15 09:40:43.502625] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.507936] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}) [2025-05-15 09:40:43.507956] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1}, leader="0.0.0.0:0") [2025-05-15 09:40:43.507971] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:43.507984] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1) [2025-05-15 09:40:43.507997] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_ls_location_service.cpp:439) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4721] nonblock get location failed(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}) [2025-05-15 09:40:43.508008] WDIAG [SHARE.LOCATION] nonblock_get_leader (ob_location_service.cpp:159) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] fail to nonblock get log stream location leader(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", cluster_id=1747291386, tenant_id=1002, ls_id={id:1001}, leader="0.0.0.0:0") [2025-05-15 09:40:43.508019] WDIAG [CLOG] get_leader_max_scn_ (palf_handle_impl.cpp:5340) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-4721] get_leader failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:43.508030] WDIAG [CLOG] is_in_sync_ (palf_handle_impl.cpp:5305) [4577][T1002_TimerWK0_LogUpdater][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4721] get_palf_max_scn failed(ret=-4721, self="127.0.0.1:2882", palf_id=1001) [2025-05-15 09:40:43.508070] WDIAG [STORAGE.TRANS] run1 (ob_standby_timestamp_service.cpp:141) [4629][T1002_STSWorker][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-4076] query and update last id fail(ret=-4076, ret="OB_NEED_WAIT") [2025-05-15 09:40:43.508388] WDIAG [SHARE.SCHEMA] get_tenant_status (ob_schema_getter_guard.cpp:8261) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-5157] tenant not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.508412] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_schema_getter_guard.cpp:8225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=24][errcode=-5157] fail to get tenant status(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.508421] WDIAG [SHARE.SCHEMA] check_tenant_is_restore (ob_multi_version_schema_service.cpp:4340) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-5157] fail to check tenant is restore(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.508432] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1134) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11][errcode=-5157] fail to check restore tenant exist(ret=-5157, tenant_id=1002) [2025-05-15 09:40:43.508444] WDIAG get_global_sys_variable (ob_basic_session_info.cpp:1044) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=12][errcode=-4029] fail get schema guard(ret=-4029) [2025-05-15 09:40:43.508453] WDIAG [SERVER] get_mem_limit (ob_mysql_request_manager.cpp:290) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=7][errcode=-4029] failed to get global sys variable(ret=-4029, tenant_id=1002, OB_SV_SQL_AUDIT_PERCENTAGE="ob_sql_audit_percentage", obj_val={"NULL":"NULL"}) [2025-05-15 09:40:43.508468] WDIAG [SERVER] check_config_mem_limit (ob_eliminate_task.cpp:75) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4029] failed to get mem limit(ret=-4029, tenant_id=1002, mem_limit=3887376629, config_mem_limit_=16777216) [2025-05-15 09:40:43.508481] INFO [SERVER] runTimerTask (ob_eliminate_task.cpp:225) [4578][T1002_TimerWK1_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=11] sql audit evict task end(request_manager_->get_tenant_id()=1002, evict_high_mem_level=8388608, evict_high_size_level=471859, evict_batch_count=0, elapse_time=0, size_used=0, mem_used=8318976) [2025-05-15 09:40:43.508525] INFO [SERVER] prepare_alloc_queue (ob_dl_queue.cpp:92) [4580][T1002_TimerWK3_ReqMemEvict][T1002][Y0-0000000000000000-0-0] [lt=0] Construct Queue Num(construct_num=0, get_push_idx()=8, get_cur_idx()=0, get_pop_idx()=0) [2025-05-15 09:40:43.508895] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] get wrs ts(ls_id={id:1}, delta=12345757113, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.508918] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289693216794002, v:0}", last_lsn={lsn:934281717}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.508939] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.508956] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=16][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.508967] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.508983] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=11] get wrs ts(ls_id={id:1001}, delta=12345757113, log_service_ts=1747289697751194, min_tx_service_ts=4611686018427387, timestamp={val:1747289697751194001, v:0}) [2025-05-15 09:40:43.508997] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=12] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1002, LS_ID={id:1001}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:18446744073709551615, v:3}", last_lsn={lsn:18446744073709551615}, last_gts={val:0, v:0}, min_start_scn="{val:18446744073709551615, v:3}", min_start_status=0, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.509011] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1001}, ls_weak_read_ts:{val:1747289697751194001, v:0}}) [2025-05-15 09:40:43.509020] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1001}) [2025-05-15 09:40:43.509030] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] handle ls fail(ls_id={id:1001}, ret=-4023) [2025-05-15 09:40:43.509044] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.509054] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1002) [2025-05-15 09:40:43.509061] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4630][T1002_TenantWea][T1002][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1002, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043508874) [2025-05-15 09:40:43.512754] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.520882] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=15][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.520903] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:43.520921] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:43.520936] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4602][T1002_LogRessvr][T1001][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.520949] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1"}, aret=-5627, ret=-5627) [2025-05-15 09:40:43.520962] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.520981] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=18][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.520998] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=16][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:43.521014] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=14][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78cb0f0, start=1747302043520863, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.521027] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:43.521042] WDIAG [SHARE] get_source (ob_restore_table_operator.cpp:127) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] sql read failed(ret=-5627, sql=select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1) [2025-05-15 09:40:43.521133] WDIAG [SHARE] get_source (ob_log_restore_source_mgr.cpp:186) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=17][errcode=-5627] table_operator_ get_source failed(ret=-5627) [2025-05-15 09:40:43.521146] WDIAG [CLOG] get_source_ (ob_remote_location_adaptor.cpp:146) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=13][errcode=-5627] get source failed(ret=-5627) [2025-05-15 09:40:43.521154] WDIAG [CLOG] update_upstream (ob_remote_location_adaptor.cpp:79) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=7][errcode=-5627] get source failed(tenant_id=1002) [2025-05-15 09:40:43.521166] WDIAG [CLOG] do_thread_task_ (ob_log_restore_service.cpp:181) [4602][T1002_LogRessvr][T1002][YB427F000001-000635291D2C6112-0-0] [lt=9][errcode=-5627] update_upstream_ failed [2025-05-15 09:40:43.522855] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=13][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.530654] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12345802622, log_service_ts=1747289697727835, min_tx_service_ts=4611686018427387, timestamp={val:1747289697727835001, v:0}) [2025-05-15 09:40:43.530680] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=23] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289694525675001, v:0}", last_lsn={lsn:2944212097}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.530682] WDIAG [STORAGE.TRANS] process_cluster_heartbeat_rpc_cb (ob_tenant_weak_read_service.cpp:446) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=9][errcode=-4076] tenant weak read service cluster heartbeat RPC fail(ret=-4076, rcode={code:-4076, msg:"post cluster heartbeat rpc failed, tenant_id=1001", warnings:[]}, tenant_id_=1001, dst="127.0.0.1:2882", cluster_service_tablet_id={id:226}) [2025-05-15 09:40:43.530702] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=20][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697727835001, v:0}}) [2025-05-15 09:40:43.530704] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:863) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=21][errcode=-4076] post cluster heartbeat rpc fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, local_server_version={val:18446744073709551615, v:3}, valid_part_count=0, total_part_count=0, generate_timestamp=1747302043530676) [2025-05-15 09:40:43.530717] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=15][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.530717] WDIAG [STORAGE.TRANS] do_cluster_heartbeat_ (ob_tenant_weak_read_service.cpp:873) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13][errcode=-4076] tenant weak read service do cluster heartbeat fail(ret=-4076, ret="OB_NEED_WAIT", tenant_id_=1001, last_post_cluster_heartbeat_tstamp_=1747302043330632, cluster_heartbeat_interval_=1000000, cluster_service_tablet_id={id:226}, cluster_service_master="0.0.0.0:0") [2025-05-15 09:40:43.530726] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=8][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.530738] INFO [STORAGE.TRANS] generate_weak_read_timestamp_ (ob_ls_wrs_handler.cpp:190) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10] get wrs ts(ls_id={id:1}, delta=12345630129, log_service_ts=1747289697900328, min_tx_service_ts=4611686018427387, timestamp={val:1747289697900328000, v:0}) [2025-05-15 09:40:43.530740] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.530750] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=9][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1) [2025-05-15 09:40:43.530756] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4332][T1_TenantWeakRe][T1][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043530629) [2025-05-15 09:40:43.530752] INFO [STORAGE.TRANS] print_stat_info (ob_keep_alive_ls_handler.cpp:226) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=13] [Keep Alive Stat] LS Keep Alive Info(tenant_id=1001, LS_ID={id:1}, Not_Master_Cnt=0, Near_To_GTS_Cnt=0, Other_Error_Cnt=0, Submit_Succ_Cnt=0, last_scn="{val:1747289695031638001, v:0}", last_lsn={lsn:5311361384}, last_gts={val:0, v:0}, min_start_scn="{val:0, v:0}", min_start_status=1, sys_ls_end_scn={val:0, v:0}) [2025-05-15 09:40:43.530768] WDIAG [STORAGE.TRANS] generate_ls_weak_read_snapshot_version (ob_ls_wrs_handler.cpp:113) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] get gts scn error(ret=-4023, max_stale_time=5000000, *this={is_inited:true, is_enabled:true, ls_id:{id:1}, ls_weak_read_ts:{val:1747289697900328000, v:0}}) [2025-05-15 09:40:43.530782] WDIAG [STORAGE.TRANS] handle_ls_ (ob_tenant_weak_read_service.cpp:677) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=14][errcode=-4023] generate ls weak read snapshot version fail(ret=-4023, ret="OB_EAGAIN", ls_id={id:1}) [2025-05-15 09:40:43.530792] WDIAG [STORAGE.TRANS] scan_all_ls_ (ob_tenant_weak_read_service.cpp:633) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=11][errcode=-4023] handle ls fail(ls_id={id:1}, ret=-4023) [2025-05-15 09:40:43.530805] WDIAG [STORAGE.TRANS] generate_min_weak_read_version (ob_weak_read_util.cpp:78) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] get gts cache error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.530815] WDIAG [STORAGE.TRANS] generate_server_version (ob_tenant_weak_read_service.cpp:311) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=10][errcode=-4023] generate min weak read version error(ret=-4023, tenant_id=1001) [2025-05-15 09:40:43.530821] WDIAG [STORAGE.TRANS] generate_tenant_weak_read_timestamp_ (ob_tenant_weak_read_service.cpp:591) [4497][T1001_TenantWea][T1001][Y0-0000000000000000-0-0] [lt=6][errcode=-4023] generate server version for tenant fail(ret=-4023, ret="OB_EAGAIN", tenant_id=1001, index=0x7f1af062c710, server_version_epoch_tstamp_=1747302043530727) [2025-05-15 09:40:43.532964] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.541776] INFO [SQL.EXE] run2 (ob_maintain_dependency_info_task.cpp:227) [4064][MaintainDepInfo][T0][Y0-0000000000000000-0-0] [lt=24] [ASYNC TASK QUEUE](queue_.size()=0, sys_view_consistent_.size()=0) [2025-05-15 09:40:43.542663] INFO do_work (ob_rl_mgr.cpp:708) [4033][rl_mgr0][T0][Y0-0000000000000000-0-0] [lt=23] swc wakeup.(stat_period_=1000000, ready=false) [2025-05-15 09:40:43.543068] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.546295] INFO try_report (ob_tc_stat.cpp:35) [4142][][T0][Y0-0000000000000000-0-0] [lt=21] QSched: |cfg: root:1,x,0 net_in:1,1250M,0 net_out:1,1250M,0 disk:1,x,0 t500m0:1,x,0 t500m0gd:1,x,0 t500m1:1,x,0 t500m1gd:1,x,0 t500m2:1,x,0 t500m2gd:1,x,0 t1m0:1,x,0 t1m0gd:1,x,0 t1m1:1,x,0 t1m1gd:1,x,0 t1m2:1,x,0 t1m2gd:1,x,0 t1001m0:1,x,0 t1001m0gd:1,x,0 t1001m1:1,x,0 t1001m1gd:1,x,0 t1001m2:1,x,0 t1001m2gd:1,x,0 t1002m0:1,x,0 t1002m0gd:1,x,0 t1002m1:1,x,0 t1002m1gd:1,x,0 t1002m2:1,x,0 t1002m2gd:1,x,0|stat: |q0: |q1: [2025-05-15 09:40:43.553175] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.556239] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.556263] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=20][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.556278] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.556292] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.556303] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.556316] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.556332] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=13][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.556346] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.556360] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=14][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.556368] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.556382] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.556399] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043556397, replica_locations:[]}) [2025-05-15 09:40:43.556416] INFO [SHARE.LOCATION] batch_renew_tablet_locations (ob_location_service.cpp:469) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=16] [TABLET_LOCATION] batch renew tablet locations finished(ret=0, ret="OB_SUCCESS", tenant_id=1, renew_type=0, is_nonblock=false, tablet_list=[{id:1}], ls_ids=[{id:1}], error_code=-4721) [2025-05-15 09:40:43.556440] WDIAG [SQL] do_close_plan (ob_result_set.cpp:828) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4006] exec result is null(ret=-4006) [2025-05-15 09:40:43.556450] WDIAG [SQL] do_close (ob_result_set.cpp:919) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9][errcode=0] fail close main query(ret=0, do_close_plan_ret=-4006) [2025-05-15 09:40:43.556478] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.556541] WDIAG [SQL.PC] common_free (ob_lib_cache_object_manager.cpp:139) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=22][errcode=0] set logical del time(cache_obj->get_logical_del_time()=11807447016, cache_obj->added_lc()=false, cache_obj->get_object_id()=15859, cache_obj->get_tenant_id()=1, lbt()="0x9510a36 0x12ec31de 0x8c51f6c 0x9142abc 0x8fb2622 0x8f7e2d2 0x8f7dba0 0x8f7da9a 0x8f7d8a6 0x1c7996cd 0x1cb26bd9 0x1aa047e5 0x90610ae 0x90158e7 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.557743] WDIAG [RPC] post (ob_poc_rpc_proxy.h:235) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0][errcode=-4122] check_blacklist failed(addr="172.19.0.2:2882") [2025-05-15 09:40:43.557764] WDIAG [RPC] call_rpc (ob_async_rpc_proxy.h:359) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=18][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, cluster_id=1747291386, tenant_id=1, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.557778] WDIAG [RPC] call (ob_async_rpc_proxy.h:293) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=15][errcode=-4122] call rpc func failed(server="172.19.0.2:2882", timeout=2000000, cluster_id=1747291386, tenant_id=1, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, group_id=9, ret=-4122, ret="OB_RPC_POST_ERROR") [2025-05-15 09:40:43.557791] WDIAG [RPC] on_timeout (ob_async_rpc_proxy.h:85) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] some error in rcode and enter on_timeout(AsyncCB::rcode_.rcode_=0) [2025-05-15 09:40:43.557801] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:288) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=8][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1747291386, addr="172.19.0.2:2882", timeout=2000000, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}) [2025-05-15 09:40:43.557814] WDIAG [RPC] wait (ob_async_rpc_proxy.h:454) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=0] execute rpc failed(rc=-4012, server="0.0.0.0:0", timeout=0, packet code=330, arg={addr:"172.19.0.2:2882", cluster_id:1747291386}, ret=0) [2025-05-15 09:40:43.557827] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:307) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=11][errcode=0] fail to get result by rpc, just ignore(tmp_ret=-4012, addr="172.19.0.2:2882") [2025-05-15 09:40:43.557839] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.557852] WDIAG [SHARE.PT] find_leader (ob_ls_info.cpp:844) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=12][errcode=-4018] fail to get leader replica(ret=-4018, ret="OB_ENTRY_NOT_EXIST", *this={tenant_id:1, ls_id:{id:1}, replicas:[]}, replica count=0) [2025-05-15 09:40:43.557862] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:132) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] leader doesn't exist, try use all_server_list(tmp_ret=-4018, tmp_ret="OB_ENTRY_NOT_EXIST", ls_info={tenant_id:1, ls_id:{id:1}, replicas:[]}) [2025-05-15 09:40:43.557876] INFO [SHARE.PT] get_ls_info_ (ob_rpc_ls_table.cpp:143) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=10] server_list is empty, do nothing(ret=0, ret="OB_SUCCESS", server_list=[]) [2025-05-15 09:40:43.557888] INFO [SHARE.LOCATION] batch_update_caches_ (ob_ls_location_service.cpp:953) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=9] [LS_LOCATION]ls location cache has changed(ret=0, ret="OB_SUCCESS", old_location={cache_key:{tenant_id:0, ls_id:{id:-1}, cluster_id:-1}, renew_time:0, replica_locations:[]}, new_location={cache_key:{tenant_id:1, ls_id:{id:1}, cluster_id:1747291386}, renew_time:1747302043557887, replica_locations:[]}) [2025-05-15 09:40:43.557953] INFO [SERVER] sleep_before_local_retry (ob_query_retry_ctrl.cpp:87) [4290][T1_TimerWK1_FreInfoReload][T1][YB427F000001-000635291D7C63BD-0-0] [lt=0] will sleep(sleep_us=100000, remain_us=713729, base_sleep_us=1000, retry_sleep_type=1, v.stmt_retry_times_=337, v.err_=-4721, timeout_timestamp=1747302044271680) [2025-05-15 09:40:43.559381] INFO [COMMON] compute_tenant_wash_size (ob_kvcache_store.cpp:1135) [3953][TimerWK0_KVCacheWash][T0][Y0-0000000000000000-0-0] [lt=26] Wash compute wash size(is_wash_valid=false, sys_total_wash_size=-44878627635, global_cache_size=382861312, tenant_max_wash_size=0, tenant_min_wash_size=0, tenant_ids_=[cnt:6, 500, 508, 509, 1, 1001, 1002]) [2025-05-15 09:40:43.563284] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=15][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.570154] INFO pn_ratelimit (group.c:653) [3956][TimerWK3_IngressService][T0][Y0-0000000000000000-0-0] [lt=7] PNIO set ratelimit as 9223372036854775807 bytes/s, grp_id=2 [2025-05-15 09:40:43.573393] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=17][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.574545] INFO [PALF] runTimerTask (block_gc_timer_task.cpp:97) [4460][T1001_TimerWK0_PalfGC][T1001][Y0-0000000000000000-0-0] [lt=9] BlockGCTimerTask success(ret=0, cost_time_us=5, palf_env_impl_={IPalfEnvImpl:{IPalfEnvImpl:"Dummy"}, self:"127.0.0.1:2882", log_dir:"/root/ob/observer/store/clog/tenant_1001", disk_options_wrapper:{disk_opts_for_stopping_writing:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, disk_opts_for_recycling_blocks:{log_disk_size(MB):1536, log_disk_utilization_threshold(%):80, log_disk_utilization_limit_threshold(%):95, log_disk_throttling_percentage(%):60, log_disk_throttling_maximum_duration(s):7200, log_writer_parallelism:3}, status:1, cur_unrecyclable_log_disk_size(MB):201, sequence:1}, log_alloc_mgr_:{flying_log_task:0, flying_meta_task:0}}) [2025-05-15 09:40:43.577350] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:43.577378] INFO [STORAGE.TRANS] statistics (ob_gts_source.cpp:62) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=25] gts statistics(tenant_id=1, gts_rpc_cnt=0, get_gts_cache_cnt=9104, get_gts_with_stc_cnt=0, try_get_gts_cache_cnt=0, try_get_gts_with_stc_cnt=0, wait_gts_elapse_cnt=0, try_wait_gts_elapse_cnt=0) [2025-05-15 09:40:43.577394] INFO [STORAGE.TRANS] operator() (ob_ts_mgr.h:180) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] refresh gts functor(ret=-4721, ret="OB_LS_LOCATION_NOT_EXIST", gts_tenant_info={v:1}) [2025-05-15 09:40:43.577414] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1001) [2025-05-15 09:40:43.577436] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1001) [2025-05-15 09:40:43.577457] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1001, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:43.577819] WDIAG [SHARE.LOCATION] check_ls_exist (ob_location_service.cpp:571) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-5157] tenant does not exist(ret=-5157, ret="OB_TENANT_NOT_EXIST", tenant_id=1002) [2025-05-15 09:40:43.577832] WDIAG [SHARE.SCHEMA] check_if_tenant_has_been_dropped (ob_multi_version_schema_service.cpp:1973) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0][errcode=-4006] local schema not inited,(ret=-4006, tenant_id=1002) [2025-05-15 09:40:43.577846] INFO [STORAGE.TRANS] refresh_gts_location_ (ob_gts_source.cpp:574) [4110][TsMgr][T0][Y0-0000000000000000-0-0] [lt=0] gts nonblock renew success(ret=0, tenant_id=1002, gts_local_cache={srr:[mts=0], gts:0, latest_srr:[mts=0]}) [2025-05-15 09:40:43.583510] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=16][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.584644] WDIAG [SHARE.SCHEMA] get_tenant_schema_guard (ob_multi_version_schema_service.cpp:1158) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C6670-0-0] [lt=9][errcode=-5627] get tenant schema store fail, maybe local schema is old(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.584664] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:915) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C6670-0-0] [lt=19][errcode=-5627] get schema guard failed(ret=-5627) [2025-05-15 09:40:43.584683] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:980) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C6670-0-0] [lt=8][errcode=-5627] failed to process record(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, record_ret=-5627, ret=-5627) [2025-05-15 09:40:43.584700] WDIAG [SQL] move_to_sqlstat_cache (ob_sql_stat_record.cpp:360) [4579][T1002_TimerWK2_LSMetaCh][T1001][YB427F000001-000635291F9C6670-0-0] [lt=14][errcode=0] the key is not valid which at plan cache mgr(ret=0, ret="OB_SUCCESS") [2025-05-15 09:40:43.584710] WDIAG [SERVER] query (ob_inner_sql_connection.cpp:1017) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=10][errcode=-5627] failed to process final(executor={ObIExecutor:, sql:"SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port"}, aret=-5627, ret=-5627) [2025-05-15 09:40:43.584723] WDIAG [SERVER] execute_read_inner (ob_inner_sql_connection.cpp:1807) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=12][errcode=-5627] execute sql failed(ret=-5627, tenant_id=1001, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.584736] WDIAG [SERVER] retry_while_no_tenant_resource (ob_inner_sql_connection.cpp:1078) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=12][errcode=-5627] retry_while_no_tenant_resource failed(ret=-5627, tenant_id=1001) [2025-05-15 09:40:43.584747] WDIAG [SERVER] execute_read (ob_inner_sql_connection.cpp:1747) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=10][errcode=-5627] execute_read failed(ret=-5627, cluster_id=1747291386, tenant_id=1001) [2025-05-15 09:40:43.584758] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:134) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=10][errcode=-5627] query failed(ret=-5627, conn=0x7f1ae78ca3b0, start=1747302043584620, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.584773] WDIAG [COMMON.MYSQLP] read (ob_mysql_proxy.cpp:63) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=15][errcode=-5627] read failed(ret=-5627) [2025-05-15 09:40:43.584785] WDIAG [SHARE.PT] get_by_tenant (ob_persistent_ls_table.cpp:639) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=9][errcode=-5627] execute sql failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, sql=SELECT * FROM __all_ls_meta_table WHERE tenant_id = 1002 ORDER BY tenant_id, ls_id, svr_ip, svr_port) [2025-05-15 09:40:43.584857] WDIAG [SHARE.PT] get_by_tenant (ob_ls_table_operator.cpp:252) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=13][errcode=-5627] get all ls info by persistent_ls_ failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002) [2025-05-15 09:40:43.584870] WDIAG [SHARE] inner_open_ (ob_ls_table_iterator.cpp:103) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=12][errcode=-5627] fail to get ls infos by tenant(ret=-5627, ret="OB_SCHEMA_EAGAIN", tenant_id=1002, inner_table_only=false) [2025-05-15 09:40:43.584883] WDIAG [SHARE] next (ob_ls_table_iterator.cpp:70) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=13][errcode=-5627] fail to open iterator(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:43.584895] WDIAG [SERVER] build_replica_map_ (ob_tenant_meta_checker.cpp:339) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=11][errcode=-5627] ls table iterator next failed(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:43.584910] WDIAG [SERVER] check_ls_table_ (ob_tenant_meta_checker.cpp:219) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=12][errcode=-5627] build replica map from ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:43.584923] WDIAG [SERVER] check_ls_table (ob_tenant_meta_checker.cpp:193) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=11][errcode=-5627] check ls table failed(ret=-5627, ret="OB_SCHEMA_EAGAIN", mode=0) [2025-05-15 09:40:43.584934] WDIAG [SERVER] runTimerTask (ob_tenant_meta_checker.cpp:41) [4579][T1002_TimerWK2_LSMetaCh][T1002][YB427F000001-000635291F9C6670-0-0] [lt=11][errcode=-5627] fail to check ls meta table(ret=-5627, ret="OB_SCHEMA_EAGAIN") [2025-05-15 09:40:43.591621] WDIAG [SHARE] refresh (ob_task_define.cpp:396) [3961][LogLimiterRefre][T0][Y0-0000000000000000-0-0] [lt=25][errcode=0] Throttled WDIAG logs in last second(details {error code, dropped logs, earliest tid}=[{errcode:-4721, dropped:412, tid:4577}]) [2025-05-15 09:40:43.593651] WDIAG [SERVER.OMT] check_cgroup_root_dir (ob_cgroup_ctrl.cpp:212) [4195][MultiTenant][T0][Y0-0000000000000000-0-0] [lt=23][errcode=-4027] dir not exist(OBSERVER_ROOT_CGROUP_DIR="cgroup", ret=-4027) [2025-05-15 09:40:43.595307] INFO [RPC.OBMYSQL] sm_conn_build_handshake (obsm_conn_callback.cpp:117) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=25] new mysql sessid created(conn.sessid_=3221553330, support_ssl=false) [2025-05-15 09:40:43.595429] INFO [RPC.OBMYSQL] init (obsm_conn_callback.cpp:137) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=20] sm conn init succ(conn.sessid_=3221553330, sess.client_addr_="172.22.0.1:40620") [2025-05-15 09:40:43.595448] INFO [RPC.OBMYSQL] do_accept_one (ob_sql_nio.cpp:1086) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=13] accept one succ(*s={this:0x7f1a1de1e030, session_id:3221553330, trace_id:Y0-0000000000000000-0-0, sql_handling_stage:-1, sql_initiative_shutdown:false, reader:{fd:145}, err:0, last_decode_time:0, pending_write_task:{buf:null, sz:0}, need_epoll_trigger_write:false, consume_size:0, pending_flag:0, may_handling_flag:true, handler_close_flag:false}) [2025-05-15 09:40:43.595683] INFO [SERVER] extract_user_tenant (obmp_connect.cpp:61) [4128][sql_nio0][T0][Y0-0000000000000000-0-0] [lt=29] username and tenantname(user_name=root, tenant_name=) [2025-05-15 09:40:43.595719] INFO [SERVER] dispatch_req (ob_srv_deliver.cpp:276) [4128][sql_nio0][T1][Y0-0000000000000000-0-0] [lt=15] succeed to dispatch to tenant mysql queue(tenant_id=1) [2025-05-15 09:40:43.595746] WDIAG [SERVER] check_server_empty (ob_service.cpp:2250) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B9-0-0] [lt=17][errcode=0] [CHECK_SERVER_EMPTY] server_id exists(GCTX.get_server_id()=1, server_id_in_GCONF=1) [2025-05-15 09:40:43.595764] WDIAG [SERVER] get_tenant_id (obmp_connect.cpp:1604) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B9-0-0] [lt=17][errcode=-8001] tenant schema not refreshed yet(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_id=1) [2025-05-15 09:40:43.595777] WDIAG [SERVER] check_update_tenant_id (obmp_connect.cpp:2227) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B9-0-0] [lt=12][errcode=-8001] get_tenant_id failed(ret=-8001) [2025-05-15 09:40:43.595790] WDIAG [SERVER] process (obmp_connect.cpp:333) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B9-0-0] [lt=12][errcode=-8001] fail to check update tenant id(ret=-8001, ret="OB_SERVER_IS_INIT", tenant_name_=) [2025-05-15 09:40:43.595815] INFO [SERVER] send_error_packet (obmp_packet_sender.cpp:373) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B9-0-0] [lt=11] sending error packet(ob_error=-8001, client error=-8001, extra_err_info=NULL, lbt()="0x9510a36 0x10da1920 0x10d4c5c6 0x916ba98 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7f1af0639dd3") [2025-05-15 09:40:43.595835] WDIAG [SERVER] get_session (obmp_packet_sender.cpp:619) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B9-0-0] [lt=14][errcode=-4018] get session fail(ret=-4018, sessid=3221553330, proxy_sessid=0) [2025-05-15 09:40:43.595874] WDIAG [SERVER] disconnect (obmp_packet_sender.cpp:862) [4323][T1_MysqlQueueTh][T1][Y0-000635291BEC90B9-0-0] [lt=13][errcode=0] server close connection(sessid=3221553330, proxy_sessid=0, stack="0x9510a36 0x10da54e0 0x10d73bcb 0x916b1ab 0x8c325f6 0x249285d2 0x9763dea 0x24929549 0x23342f45 0x23340b2e 0x7f1af0a081cf 0x7